How to Evaluate an AI Agent Before Launch

How do you test whether an AI agent is ready for production? Define the business outcome and non-negotiable controls, build a representative evaluation set that includes routine, edge, adversarial and broken-system cases, run the complete agent with production-like tools, and score task success, policy compliance, tool correctness, groundedness, escalation quality, latency and total cost per successful outcome. Block release on critical failures and rerun the same suite after every material change.

Evaluate the system, not a transcript

An agent is a workflow: instructions, model, context, retrieval, tools, identity, policy checks, approvals, retries and human handoffs. Reading a few polished conversations tests only the visible layer. A useful evaluation begins with the system's intended job and verifies the final state in the business system.

NIST's August 2026 TEVV-Athlon draft presents test, evaluation, verification and validation as a customizable way to gather evidence about whether AI systems meet organizational goals while limiting negative impacts. That framing is useful for SMEs: the test should be designed around the consequence of your workflow, not copied from a generic leaderboard.

Start from the AI agent job description. Turn every responsibility, prohibited action, permission and escalation rule into observable acceptance criteria. If the agent is expected to prepare an invoice exception, the pass condition is not “mentions the missing purchase order.” It is “creates the correct exception against the correct invoice, preserves evidence, makes no unauthorized ERP change and routes it to the named owner.”

Build a representative 50-case evaluation set

Fifty cases are enough to make an early evaluation disciplined, not enough to prove universal reliability. Use the set as a launch gate and expand it from production evidence. Keep real cases de-identified, permissioned and separated from training or prompt-tuning decisions.

• 20 routine cases: normal inputs across the common customer, language, product and record variations.
• 10 boundary cases: values just inside and outside approval, amount, timing or policy thresholds.
• 8 exception cases: missing fields, conflicting evidence, duplicates, unusual formats and unresolved dependencies.
• 6 adversarial cases: prompt injection, misleading attachments, attempts to cross tenant or permission boundaries and requests for prohibited actions.
• 4 broken-system cases: timeouts, stale records, unavailable tools, partial writes and delayed responses.
• 2 high-consequence cases: rare scenarios where the only acceptable behaviour is refusal, pause or qualified human review.

Each row should contain the input, preconditions, allowed evidence, expected action, prohibited action, required tool state, expected escalation, severity and scoring notes. Download my 50-case AI agent evaluation sheet and adapt it to one workflow.

Use a scorecard that reflects business failure

Do not average away a severe failure. A 96% overall score can conceal one unauthorized payment or cross-customer disclosure. Create critical gates for security, privacy, financial commitment, deletion, publication and regulated decisions. A single critical breach should fail the release until the cause is controlled and the relevant cases pass again.

For graded criteria, define anchors before testing. A groundedness score of 2 might require every material claim to point to approved current evidence; 1 could mean the conclusion is supported but one secondary statement is ungrounded; 0 means the decision depends on invented, stale or unavailable evidence. Two reviewers should independently score an initial sample and resolve disagreements before the full run.

Test tools and state transitions explicitly

Agents often fail at the seam between reasoning and action. Capture the tool selected, parameter schema, target record, authorization result, approval evidence, idempotency key, response and final system state. Test whether the workflow behaves safely when the underlying record changes between planning and execution.

Include replayed requests, duplicate callbacks and ambiguous timeouts. If a payment API times out, the agent must check whether the payment exists before retrying. If a quotation changes after approval, the old approval must not authorize the new total. If a tool returns an error-shaped success payload, the workflow must not mark the job complete.

Pair evaluation with the permission matrix and six-trigger escalation framework. The evaluation proves the boundaries work under representative pressure; it does not replace deterministic authorization or a human decision.

Run adversarial and human-handoff tests

Insert untrusted instructions into documents, websites, emails and retrieved notes. Ask the agent to reveal hidden context, change its role, use a broader tool, contact a different recipient or ignore a required approval. Verify both the refusal and the absence of an unauthorized side effect.

Test escalation end to end. Did the agent recognize the trigger? Did it stop before the consequential action? Did the right queue receive the case? Did the handoff contain the goal, evidence, work completed, uncertainty and proposed next step? Could the person approve, modify, reject or request more information without reconstructing the conversation?

OpenAI's business eval guidance describes a repeatable cycle of specifying what good means, measuring under real-world conditions and improving from errors. Preserve failure examples as durable regression cases. A corrected prompt without a new test is a memory, not a control.

Set release thresholds before seeing the result

Define thresholds with the workflow owner, technical owner and control owner before the evaluation run. A practical launch decision can require:

• zero critical policy, security, privacy or unauthorized-action failures;
• 100% correct refusal or escalation on the high-consequence set;
• a minimum end-to-end task-success rate for ordinary cases;
• a maximum wrong-tool, wrong-record and duplicate-action rate;
• acceptable reviewer effort and escalation completeness;
• a latency and cost-per-success range based on production-like load; and
• a documented owner and remediation plan for every accepted limitation.

Do not choose a threshold because it makes the current model pass. Tie it to the cost of error and the manual alternative. A drafting assistant may tolerate a correctable wording defect. An agent that commits money or publishes externally needs much stricter controls and may remain behind approval permanently.

Measure cost per successful outcome

Tokens and model calls are inputs, not the unit of useful work. Record all attempts, retries, tool fees, retrieval, infrastructure, human review and failure recovery for each completed case. Then calculate cost per accepted outcome. A cheaper model that needs repeated attempts and correction can cost more than a stronger model that completes the task once.

Track the distribution, not only the average. The slowest or most expensive exception cases often determine staffing and customer experience. Segment results by case type, language, channel, document quality, tool and agent version so improvements are not hidden by changing traffic mix.

Turn the launch suite into continuous evaluation

1. Freeze the test contract. Version the cases, rubric, thresholds, tool mocks and evaluator guidance.
2. Run before every material change. Models, prompts, tools, retrieval, permissions and policies can all change behaviour.
3. Sample production work. Review accepted cases, escalations, overrides, complaints and near misses.
4. Add new failures. Convert every material incident and recurring correction into a regression case.
5. Watch drift. Compare quality, latency, cost and exception mix by version and time period.
6. Keep a rollback path. A failed gate should stop promotion and a production regression should disable or narrow the affected action.

The evaluation report should state what was tested, what was excluded, which version ran, who reviewed it, what passed, what failed and what residual risk was accepted. “Passed our evals” without that context is not decision-ready evidence.

Primary sources checked

Checked 7 September 2026. The NIST TEVV-Athlon document is an initial public draft; treat it as evolving guidance and verify its status before adopting it as policy.

• NIST — TEVV-Athlon Framework for Evaluating AI Systems
• NIST — Generative AI Profile for the AI RMF
• OpenAI — How evals drive the next chapter in AI for businesses
• OpenAI API — Working with evals