AI Governance Has a Control Problem, Not a Policy Problem
Autonomy is scaling faster than assurance
Consider a relatively simple AI agent.
It might be able to:
- Read information from internal systems
- Search documents and databases
- Make decisions based on predefined criteria
- Trigger workflows
- Create or modify records
- Call external services
- Generate and send communications
- Potentially initiate actions without a human reviewing every step
The question isn’t whether the organisation has an AI policy saying that these activities should be controlled.
The question is whether the organisation can demonstrate that they are controlled.
There is an important distinction.
“AI must be used responsibly” is not a control.
A control is something you can test, monitor and prove is working.
That distinction matters enormously when the thing being controlled is capable of taking action autonomously.
The four questions I would start with
For any AI capability that can access information or take action, I think there are four fundamental questions.
1. What can it access?
What data can the AI agent see?
Which systems can it connect to?
What identities and credentials does it use?
Can it access sensitive information?
Does it have access to everything the user has access to, or only what it actually needs?
And perhaps most importantly:
Can we demonstrate that access is restricted to what was intended?
This is where traditional security controls such as identity, least privilege, segregation and access management remain critically important.
AI doesn’t make those controls less relevant.
It makes them more important.
Because an agent with excessive permissions can potentially operate at machine speed.
2. What can it change?
Reading information and changing information are not the same risk.
An agent that retrieves a document is fundamentally different from one that can:
- Change a customer record
- Modify a financial transaction
- Alter a configuration
- Create an account
- Approve something
- Delete information
- Deploy code
- Trigger an operational process
This is why I keep coming back to a simple principle:
Read fast, write slow.
Let agents move quickly where the risk is low.
But when an AI capability can change state, the control requirements should become progressively stronger.
That might mean additional approval, tighter permissions, transaction limits, validation rules, human oversight or stronger monitoring.
The objective isn’t to put a human in front of every AI action.
That simply doesn’t scale.
The objective is to put the right control around the right level of autonomy.
3. Who approved that capability?
This is an area I think deserves much more attention.
It’s easy to ask whether an AI system has been approved.
It’s harder to ask:
What exactly was approved?
Was the approval for the model?
The application?
The use case?
The data?
The tools it can call?
The actions it can perform?
The level of autonomy?
The specific permissions?
These are not necessarily the same thing.
An organisation might approve an AI assistant for a particular business process and subsequently give it access to additional systems or capabilities.
At that point, the original approval may no longer represent the actual risk.
So governance needs to establish a clear relationship between:
Capability → Permission → Owner → Approval → Risk
If nobody can clearly explain who approved an AI agent having a particular capability, that’s a governance problem.
And if nobody owns the decision, it becomes very difficult to assure.
4. What evidence shows it stayed inside those boundaries?
This is the question I find hardest.
And arguably the most important.
It isn’t enough to know what an AI agent was supposed to do.
We need to know what it actually did.
That means being able to answer questions such as:
- What data did it access?
- What decisions did it make?
- What tools did it call?
- What actions did it take?
- What changes did it make?
- What permissions did it use?
- Were any actions outside the expected pattern?
- Were exceptions detected?
- Who intervened?
- What happened when a control failed?
This is where governance moves from policy into assurance.
Because ultimately:
If you can’t produce evidence of what an autonomous system did, how can you provide assurance that it stayed within its authorised boundaries?
Evidence needs to become part of the design
This is where I think organisations need to change their approach.
Evidence shouldn’t be something we try to collect after deployment.
It should be designed into the AI capability from the beginning.
If an agent is allowed to take an action, we should already know what evidence that action will generate.
For example:
Access
We should be able to demonstrate what the agent could access and what it actually accessed.
Decision
We should be able to understand the decision or rule that resulted in an action, within the limits of what the technology can reliably explain.
Action
We should have an auditable record of what the agent actually did.
Approval
We should know who authorised the capability and under what conditions.
Exception
We should know when the agent moved outside expected behaviour and what happened next.
That turns AI governance into something much more tangible.
Not just:
“Do we have an AI policy?”
But:
“Can we demonstrate that the AI operated within the control boundaries we established?”
Don’t treat every AI action the same
One of the mistakes we could make is applying the same control model to every AI capability.
That won’t work.
There is a significant difference between an agent reading publicly available information and an agent making a change to a critical business system.
The control should reflect the potential impact.
A useful way of thinking about it is:
Low-risk read → higher autonomy
About Author
