When an AI Agent Should Not Be Fully Autonomous

Which decisions should an AI agent never make alone? An agent should not act alone when a wrong action could materially harm a person, commit money or reputation, expose sensitive data, change access, create a legal obligation, or be difficult to reverse. It should also stop when its evidence is incomplete, instructions conflict, the situation falls outside tested boundaries, or repeated attempts fail. In those cases, the safe output is a reviewable proposal with sources, assumptions and the exact action awaiting approval.

The useful question is not “Can the model do this?” It is “What authority can this workflow safely delegate?” An agent combines a model with instructions and tools. Once those tools can send, update, approve, purchase, delete or disclose, a plausible answer can become a real-world side effect. That is why autonomy must be assigned per action rather than per agent.

A sales-research agent, for example, may independently read permitted public sources, summarize an account and create an internal draft. That does not justify letting it promise a discount, email an unverified claim or change the opportunity owner. The same agent can operate at different autonomy levels within one workflow.

A five-level AI agent autonomy ladder

Level 1: observe

The agent can retrieve permitted information, organize it and produce a traceable summary. It cannot change a system of record or communicate externally. Examples include classifying support tickets, finding missing invoice fields, comparing a policy with a request, or preparing an account brief. This is the right starting point when the workflow is new, the data is sensitive, or output quality is not yet measured.

Level 2: propose

The agent prepares a draft decision or action, but an accountable person reviews the evidence and performs—or explicitly approves—the commit. Examples include a refund recommendation, a proposed CRM update, a reply draft, an expense-code suggestion or a remediation plan. The reviewer must see enough context to disagree; a decorative “approve” button after the action is already determined is not meaningful oversight.

Level 3: act inside narrow boundaries

The agent may execute low-impact, reversible operations governed by deterministic rules. It might tag a ticket, assign a task within one team, schedule an internal reminder, or update a non-sensitive field from a verified source. Boundaries should cover the allowed records, fields, values, volume, time window and maximum number of retries. Anything outside them becomes a Level 2 proposal.

Level 4: supervised by exception

The agent completes a bounded workflow while monitoring confidence, policy and operational signals. It pauses on missing evidence, an unusual amount, a sensitive category, conflicting records, a permission failure or a threshold breach. This level requires dependable detection of the conditions that make the case unusual. If the system cannot recognize its own boundary, exception-based supervision becomes unattended risk.

Level 5: open-loop autonomy

The agent plans and acts without routine human review. Use this only where consequences are low, the operating domain is constrained, failures are quickly detectable, recovery is reliable and the organization accepts the residual risk. “Fully autonomous” should not mean unrestricted access. Identity, least privilege, rate limits, logs, budgets, stop conditions and incident ownership still apply.

Five factors that should reduce autonomy

Score each proposed action—not the product as a whole—against the following factors. A single severe factor can justify a mandatory approval even if the other four are low.

This is a prioritization framework, not a legal classification. Applicable duties depend on the system, sector, jurisdiction and role. For example, Article 14 of the EU AI Act requires high-risk AI systems to be designed for effective human oversight and describes capabilities such as understanding limitations, detecting anomalies, avoiding over-reliance, overriding output and stopping the system. A business should obtain qualified advice for its actual use case rather than treating a generic checklist as compliance.

Actions that normally need a person in the loop

• Financial commitments: making payments, changing bank details, approving expenses, issuing large refunds, changing prices or accepting commercial terms.
• External commitments: publishing, sending sensitive messages, making promises, agreeing deadlines or representing a person's judgment.
• Identity and access: adding users, changing roles, issuing credentials, disabling controls or granting access to confidential records.
• People decisions: hiring, discipline, performance, compensation, eligibility or other decisions that materially affect a person.
• Destructive changes: deleting records, overwriting evidence, cancelling orders, closing accounts or triggering an irreversible downstream process.
• Unresolved exceptions: cases with conflicting sources, insufficient evidence, policy ambiguity, repeated failure or an untested pattern.

Human review is not automatically safe. The reviewer needs authority, time, relevant evidence and a realistic way to reject the recommendation. NIST's AI Risk Management Framework emphasizes clearly defined human roles and responsibilities across the AI lifecycle; the EU AI Act explicitly warns against automatically relying on system output. Design the review task, not only the review button.

Approval patterns that preserve flow

Oversight does not require a person to reperform every step. It requires placing judgment where consequences change.

1. Draft → review → send. The agent assembles context and a message; the owner verifies recipients, claims and commitment before release.
2. Propose → confirm → commit. The agent shows the exact record, current value, proposed value and reason; a separate authorized operation applies the approved change.
3. Threshold approval. Routine cases stay within pre-agreed limits; amounts, categories, confidence or risk signals above the limit stop for review.
4. Two-person control. Separate preparation and approval for the most sensitive financial, access or rights-affecting actions.
5. Sampled assurance. Review a meaningful sample of low-risk completed work and all exceptions, then use findings to update tests and boundaries.

The SOP-to-AI workflow guide shows how to separate deterministic rules, AI interpretation, exceptions and approvals. The agent comparison guide helps decide whether an agent is needed at all.

A kill switch is a recovery system, not a red button

A visible pause control matters, but it is only one part of stopping safely. Before launch, define how an owner will:

• stop new jobs and freeze queued actions;
• revoke or rotate the agent's credentials and tool access;
• identify every action taken within the affected window;
• reverse versioned changes or contain irreversible ones;
• preserve evidence without exposing secrets or sensitive content;
• notify the process owner, security contact and affected teams;
• relaunch only after the failure is reproduced, fixed and tested.

Set automatic stop conditions as well: retry limits, rate limits, spending caps, unusual tool sequences, permission denials, data-class violations and output-validation failures. OpenAI's agent-building guidance identifies repeated failures and high-risk actions as two primary triggers for human intervention. OWASP's agentic threat work likewise treats tools, identity, memory and human oversight as connected attack surfaces rather than isolated controls.

Apply the ladder to one real workflow

List every action the agent may take. For each one, write the maximum credible consequence, evidence required, reversibility, data class, applicable policy, allowed tool scope, automatic stop conditions and accountable owner. Assign a starting autonomy level. Then build test cases for normal work, missing inputs, hostile content, duplicate requests, permission failures, tool outages and borderline thresholds.

Start one level lower than your first instinct. Measure task success, false approvals, false blocks, exception quality, recovery time and human correction. Raise autonomy only for the specific action whose evidence supports it; do not promote the entire agent because one part performs well.

The MCP business guide covers how to expose narrow tools and enforce authorization at the integration boundary. Human oversight sits above that boundary: it decides when a permitted action should still wait for accountable judgment.

Primary sources checked for this guide

Checked 25 August 2026. These official and standards-oriented sources informed the oversight, intervention, security and legal-boundary guidance above.

• NIST — Artificial Intelligence Risk Management Framework
• NIST AI 600-1 — Generative AI Profile
• EUR-Lex — Regulation (EU) 2024/1689, Article 14 human oversight
• OpenAI — A practical guide to building AI agents
• OWASP GenAI Security Project — Agentic Threats Navigator