What should a business monitor after deploying an AI agent? Monitor five connected layers: business outcomes, agent decisions, tool actions, model and retrieval behaviour, and system health. Correlate them with one workflow-run ID; alert on unsafe actions, missing approvals, failed or duplicated writes, rising exceptions and breached response commitments; and review corrections, drift, cost per successful outcome and unresolved incidents on a fixed cadence.
Observability is the ability to explain a run
A dashboard with uptime, request count and tokens does not tell an operations manager whether the right invoice was routed or the customer received a valid answer. Agent observability must connect technical telemetry to the business state before and after the run.
OpenTelemetry describes observability through traces, metrics and logs, and its generative-AI work standardizes attributes for model and agent operations. Those standards can make signals portable across tools. They do not decide which business outcome matters, what content is safe to record or which alert deserves a person at 2 AM. The workflow owner must define those.
NIST's 2026 paper on monitoring deployed AI systems notes practical challenges in detecting changes after launch. Treat observability as a designed control: decide what evidence will prove normal operation, which deviations require containment and who owns the response.
The five-layer monitoring model
| Layer | Record | Question answered |
|---|---|---|
| Business outcome | Completed work, accepted output, cycle time, backlog, quality and customer or financial result | Owner sees whether the agent creates value |
| Agent decision | Route chosen, evidence references, policy result, uncertainty, refusal and escalation | Reviewer can reconstruct why work moved |
| Tool action | Tool, target, parameters, approval, idempotency key, response and final state | Operator can verify what changed |
| Model and retrieval | Model/version, calls, latency, usage, retrieved sources and grounding result | Technical owner can diagnose variation |
| System health | Queue age, timeouts, retries, dependency errors, rate limits and fallback use | Team can respond before cases accumulate |
Use a correlation ID from the workflow trigger through every model call, retrieval, policy check, approval, tool action, queue event and human correction. Add the agent version, workflow version and tenant or business unit as safe structured fields. That lets a team move from “success dropped” to the exact version, case type and dependency involved.
Do not store secrets, unrestricted prompts or full sensitive records merely because a tracing product can capture them. Record references, hashes, classifications and redacted excerpts where they provide enough evidence. Separate the operational log from access-controlled content needed for a formal investigation.
A minimal SME dashboard
A small team does not need dozens of charts. Start with one page that answers six operating questions:
- Useful work: eligible cases, completed cases and accepted business outcomes.
- Quality: pass rate from sampled review, correction rate, repeat contact, rework and confirmed incidents.
- Boundaries: refusal and escalation rate by trigger, missed escalations and approval outcomes.
- Actions: successful, failed, retried and duplicate-prevented tool operations by consequence class.
- Service: end-to-end cycle time, queue age, human response target and customer commitment breaches.
- Economics: total operating cost and cost per accepted outcome, including review and recovery.
Show rate and count together. A 50% failure rate may represent one of two cases; a 1% critical error could represent hundreds. Include the denominator, comparison period, agent version and traffic mix so apparent drift is not simply a new language, channel or exception category.
Trace decisions and tools without exposing everything
For each consequential action, capture the initiating user or schedule, agent identity and version, approved purpose, target record, requested action, effective permission, policy result, approval identity, validated parameters, idempotency key, tool response and confirmed final state. Keep source references that support the decision.
Log the reason for a refusal or escalation as a controlled code plus a short factual note. Free-form explanations alone are difficult to aggregate. Controlled codes alone can hide context. Use both, with a link to the original case where authorized.
OpenAI's API guidance recommends logging request IDs and supports client-supplied request IDs for troubleshooting. Provider request IDs should sit underneath your own workflow-run ID; one business case can span several provider requests, tools and retries.
Alert on conditions that require action
An alert should name an owner and a safe next step. Avoid paging on every model refusal or transient retry. Good immediate alerts include an unauthorized-action attempt, cross-tenant access signal, missing mandatory approval, confirmed duplicate write, unexplained external send, sensitive-data disclosure, security event or failure of the kill path.
Operational alerts can use short windows and minimum counts: a spike in failed tool calls, rising queue age, repeated fallback activation, a breached customer promise, or cost per success outside its control range. Quality and drift often need scheduled review because labels arrive after human sampling or downstream correction.
Route alerts according to consequence. The workflow owner handles backlog and service issues; the technical owner handles dependency and version failures; security or privacy owners handle exposure; and a business approver handles policy exceptions. Define the manual fallback before launch.
Review corrections and overrides as evidence
Every human modification is a potential signal, but not every human decision is correct. Capture approve, modify, reject, request-more-evidence and override outcomes with structured reasons. Sample non-escalated cases too; otherwise the dashboard sees only risks the agent already recognized.
Feed recurring corrections into the agent evaluation set. A new supplier format, regional language pattern, policy exception or tool timeout should become a regression case. Use incident review to decide whether to change data, tools, workflow rules, permissions, prompts, the model or the business process.
Detect drift with stable comparisons
Freeze a small reference set and run it against every material version. In production, compare like with like by workflow, channel, language, customer group and case difficulty. Watch task success, critical failures, correction and escalation rates, latency, tool retries and cost per accepted outcome.
Investigate both sudden and gradual changes. A sudden drop may follow a model, tool, policy or data-source release. Gradual drift may come from changing customer behaviour, document formats, knowledge freshness or staff workarounds. Maintain a change ledger so operators can overlay releases and incidents on metric trends.
A practical operating cadence
- Continuous: critical security, approval, unauthorized action and irreversible-write alerts.
- Daily: failed or stuck cases, queue age, breached promises, unusual cost and unresolved incidents.
- Weekly: sampled quality, corrections, escalations, tool failures, cost per success and new regression cases.
- Monthly: business outcomes, drift, permissions, knowledge freshness, accepted limitations and stop/scale decision.
- After every material incident: contain, preserve evidence, restore safely, find the control gap, add tests and verify the fix.
Keep the cadence proportionate. A low-risk internal drafting aid may need weekly review. A customer-facing agent with write access may require same-day oversight and immediate action on critical signals. Link the cadence to the agent's autonomy level and consequence.
Primary sources checked
Checked 7 September 2026. OpenTelemetry's generative-AI conventions continue to evolve; verify the current stability level and privacy implications before adopting fields.
- OpenTelemetry — Inside the LLM Call: GenAI Observability
- OpenTelemetry semantic conventions
- NIST — Challenges to the monitoring of deployed AI systems
- OpenAI API — request IDs and production troubleshooting
See the work, decisions and consequences
Design a minimal AI agent dashboard
I help SMEs connect traces and logs to accepted outcomes, operational alerts, review queues and improvement decisions.
Book an AI and operations audit