An agent is not safe because it asks for approval occasionally. It needs explicit thresholds for when to continue, when to ask, when to stop, and who receives the case.
An AI agent can do more than generate an answer. It can choose a tool, retrieve data, update a record, send a message, make a booking, or coordinate several steps toward a goal.
That makes the escalation policy part of the product. Without one, the agent has two bad options: stop so often that the human becomes a confirmation machine, or continue through uncertainty and convert a plausible guess into a real action.
The right balance does not come from one confidence threshold. It comes from consequence, evidence, reversibility and authority.
Escalation is a designed handoff
A complete escalation has five parts: a trigger, a destination, an evidence bundle, a deadline, and a safe waiting state. “Ask a human” defines none of them.
If an agent finds conflicting delivery addresses, it should not send a vague message to a general channel while continuing the order. It should stop the affected transaction, preserve completed reversible work, identify the conflicting fields, cite both sources, route the case to the authorised owner, set a response deadline, and prevent duplicate processing.
The agent has not passed because it stopped. It has passed when the right person can resolve the case without reconstructing it.
The escalation matrix
Four variables.
Consequence. C1 minor inconvenience. C2 recoverable operational or customer impact. C3 material financial, contractual, privacy or reputational impact. C4 safety, rights, employment, health or other critical impact.
Reversibility. R1 fully reversible before anyone else is affected. R2 reversible with small effort. R3 difficult, costly or visible to reverse. R4 effectively irreversible. An email may be impossible to recall even if the underlying record can be corrected.
Evidence state. E1 all required approved evidence present and consistent. E2 present with a minor non-material gap. E3 material evidence missing, old or conflicting. E4 source authenticity or permission uncertain.
Scope state. S1 explicitly within scope. S2 familiar variation covered by a documented rule. S3 exception without a rule. S4 prohibited or potentially malicious.
The agent may continue only inside the combination approved for that workflow.
A default decision table
| Condition | Response |
|---|---|
| C1, R1, E1, S1 | Act and log if authority permits |
| C1 to C2, R1 to R2, E1, S1 to S2 | Prepare or act within limit, then notify |
| C2 to C3 or R3 | Require approval before action |
| E3 | Request evidence or escalate exception |
| E4 | Stop, restrict access, escalate security or ownership review |
| S3 | Escalate to decision owner |
| S4 | Refuse action, log, escalate if appropriate |
| C4 or R4 | Reserved for a competent human process unless specifically assessed and authorised |
This is a starting pattern, not a universal legal rule.
Four escalation routes
Clarification. The user’s intent or input is incomplete. The agent asks one precise question and retains the current state.
Approval. The action is allowed but human confirmation is required. The approval screen must show the exact action, recipient or target, evidence, consequence, cost or limit, and what cannot be undone. “Approve agent plan” is too abstract.
Exception. The case falls outside documented rules but is not an incident. The decision owner receives the normal rule, the observed exception, evidence, options, and reversible work already completed. The resolution updates the exception catalogue. Repeated exceptions may justify a new rule, but they should not become a silent habit.
Incident. The agent may have exceeded permission, exposed sensitive information, followed injected instructions, used an unauthorised tool, duplicated an action, produced material harm or become uncontrollable. Stop relevant actions, revoke or narrow credentials, preserve logs, notify the responsible owner, activate the response procedure.
Do not ask the agent that exceeded authority to investigate itself as the only control.
Confidence is supporting evidence, not authority
Model confidence can be unavailable, poorly calibrated, or unrelated to business consequence. An agent can be highly confident and wrong. It can also be uncertain about wording while the underlying action is safe.
Use confidence only alongside evidence completeness, rule match, action consequence, reversibility and historical evaluation. Never allow a self-reported “I am 95 percent confident” to increase permission by itself.
Design the waiting state
An escalation creates unfinished work. Define how long the case can wait, whether related actions are blocked, who can reassign it, what happens at expiry, how the user sees status, and whether safe work can continue.
An agent preparing a campaign launch may continue organising approved assets while a budget exception waits. It should not publish the campaign.
The waiting state prevents two common failures: the agent continues because nobody answered, and the workflow disappears into a queue.
Avoid approval fatigue
If people approve hundreds of low-risk actions, review becomes ceremonial. Reduce the load by narrowing scope, using deterministic rules where possible, grouping similar low-risk actions, sampling stable cases, escalating only material changes, improving input quality, and measuring override and correction.
Do not solve approval fatigue by removing approval. If a human routinely approves without inspecting evidence, redesign the system or lower its authority.
Test escalation before testing autonomy
Build an escalation test set: missing required field, conflicting approved sources, expired policy, request above limit, new exception, prompt injection inside a document, unauthorised user, tool failure after partial completion, duplicate trigger, unavailable owner.
Measure correct route, evidence completeness, time to human understanding, duplicate-action prevention, safe waiting state and recovery.
Worked example: appointment rescheduling
The agent may reschedule an internal meeting when every participant authorised the behaviour, the new time stays within working hours, no external guest is involved, no protected focus block is crossed, the change occurs more than 24 hours ahead, and there is one unambiguous alternative.
It asks for approval when an external participant is involved, the meeting is inside 24 hours, the new time falls outside a participant’s normal hours, or more than one business-critical event conflicts.
It stops and escalates when calendars disagree about time zone, a participant’s permission is missing, the meeting carries a confidential or regulated label, or the calendar tool reports a partial update.
The task is simple. The escalation design is still specific.
Agent security raises the stakes
OWASP identifies prompt injection, sensitive-information disclosure and excessive agency among the major risks in language-model applications. Its Agentic Top 10 adds tool misuse and exploitation, identity and privilege abuse, memory and context poisoning, and insecure inter-agent communication.
The practical response is not to ban agents. It is to give them least privilege, narrow credentials, explicit action limits, independent validation, complete logs, reliable escalation and rapid revocation.
How we apply it
We require an escalation matrix before an agent receives any state-changing tool, reviewed by the four owners named in the AI operating model.
Every material action needs a clear answer to four questions: what is the consequence, can it be reversed, is the evidence complete, and is the case inside scope? Which authority level the agent starts from is set by automation, assistance and delegation.
Autonomy is not the absence of humans. It is a carefully defined distance between human decisions.
If you are about to give an agent a tool that changes something, tell us what it can reach and we will help you write the stop conditions first.
Sources
NIST AI Risk Management Framework Core.
NIST Generative AI Profile.
OWASP Top 10 for Agentic Applications.
OWASP Top 10 for Large Language Model Applications.
OWASP Top 10 for Model Context Protocol.
CISA and NCSC, Guidelines for Secure AI System Development.
Current to 30 July 2026. This article provides an operating framework, not legal advice.
