Vendor Questions Before Buying an AI Agent

What should you ask an AI-agent vendor before signing? Ask for evidence about the exact job, data flows, model and subprocessor use, action-level permissions, evaluation results, logs, human escalation, incident recovery, pricing under real volume, portability and deletion. A polished demo is not proof that the system can operate safely inside your business.

Start with the problem and acceptance test

Write the job before evaluating products: trigger, inputs, permitted sources, outputs, systems, allowed actions, exclusions, escalation and success measures. Require every vendor to respond to the same scenario and failure cases. Otherwise you compare presentation skill, not suitability.

Download the 30-question AI-agent vendor scorecard and score evidence as 0 = absent, 1 = stated, 2 = demonstrated, or 3 = contractually committed.

Questions 1–5: product fit

1. Which exact business outcome and user group is the agent designed for?
2. Which inputs, languages, document formats and edge cases are supported?
3. What is explicitly outside scope, and how is that boundary enforced?
4. Can the vendor run our representative normal, messy and adversarial cases?
5. Which customer evidence is independently verifiable and comparable to our workflow?

Questions 6–10: data and model supply chain

6. Map every data store, processing location, model provider and subprocessor.
7. Is customer data used to train or improve shared models, and how is that choice controlled?
8. What are the retention, backup, deletion and legal-hold rules?
9. How are prompts, retrieved documents, recordings and tool outputs protected?
10. How will the vendor notify us when a model, subprocessor or data location changes?

Questions 11–15: identity, permissions and actions

11. Does the agent use a distinct service identity rather than a shared human account?
12. Can access be limited by system, record, field, action and time?
13. Which actions require human approval, and can we configure that boundary?
14. How are secrets stored, rotated and revoked?
15. Can we immediately disable the agent without disabling the underlying business system?

Questions 16–20: evaluation and human control

16. Which task-success, policy, groundedness and tool-use measures are reported?
17. What acceptance threshold was set, on what test set, and by whom?
18. How are critical failures separated from average performance?
19. How does the system express uncertainty and transfer complete context to a human?
20. What regression testing occurs after model, prompt, tool or knowledge changes?

Questions 21–25: operations, incidents and evidence

21. Can we export an immutable record of inputs, decisions, approvals and actions?
22. Which health, quality, cost and business-outcome signals can we monitor?
23. What are the severity levels, response times and named incident contacts?
24. How are wrong system updates corrected or compensated?
25. What uptime, recovery and support commitments exclude model quality—and what covers it?

Questions 26–30: cost, ownership and exit

26. Show pricing for our volume, peak load, tools, storage, support and human review.
27. Who owns workflow logic, prompts, connectors, evaluation cases and generated records?
28. Can we export data, configurations, logs and knowledge in usable formats?
29. What happens to service and data if a provider, model or integration is withdrawn?
30. How is verified deletion completed at termination, including subprocessors and backups?

Use weighted gates, not a simple total

A vendor should not compensate for weak security or no exit path with an attractive interface. Mark non-negotiable gates before scoring. For an agent that can write to finance or customer systems, missing action-level permissions, incident evidence or exportability should stop procurement until resolved.

NIST SP 1326 frames due diligence around provenance, resilience, foundational cyber practices and supply-chain tiers. The AI RMF adds lifecycle governance, measurement and third-party contingency. Convert those concepts into evidence requests, contract commitments and a named internal owner.

Ask for an evidence room, not another presentation

Before a paid pilot, request a small evidence room. It should contain the service architecture, data-flow diagram, subprocessor list, security overview, permission model, evaluation method, incident procedure, service-status history, export formats, change policy and standard contract schedules. Sensitive reports may require a confidentiality agreement or controlled review, but “we cannot share anything” is itself procurement information.

Check consistency. The sales claim, technical documentation, security answer and contract must describe the same system. If the demo says data is never retained but the processing schedule permits broad retention, resolve the conflict in writing. If the architecture shows a model provider absent from the subprocessor list, pause.

Run a scenario-based demonstration

Give every shortlisted vendor the same representative pack. Include an ordinary case, missing information, conflicting evidence, a prompt-injection attempt, a request outside policy, an unavailable integration, a duplicate event and a customer asking for a human. Ask the vendor to show the complete trace: inputs, retrieved sources, decision, tool call, approval, result and audit record.

Do not coach the system through the demonstration more than a real user would. Record where vendor staff intervene behind the scenes. Human service is not inherently bad, but it must be included in the operating model, price and service commitment.

Test claims in your environment

A pilot should use a bounded non-production or safely isolated environment first. Configure your own permission scope, sample data and acceptance cases. Measure end-to-end accepted outcomes, not a cherry-picked model answer. Confirm that revocation works, that logs are visible to your team and that a failure does not leave a half-completed business record.

For agents connected to email, CRM, ERP or support tools, test idempotency and recovery. Repeat the same event. Interrupt the tool after it starts. Remove a permission. Change the underlying record between proposal and approval. A production agent must recognise that the world changed.

Separate vendor assurances from independent evidence

Classify evidence as stated, demonstrated, independently assessed or contractually committed. Certifications and audit reports can support confidence in the vendor's management system, but they do not prove your exact workflow is accurate, permitted or recoverable. Product documentation may prove a control exists; your configured test must prove that it works for your tenant and role.

Ask what evidence remains available after purchase. A procurement workshop is not enough if your operations team cannot export action logs or reproduce an incident. Define an evidence owner and review date inside the business.

Examine the model and tool change policy

Find out whether the vendor can change the model, system prompts, retrieval method, safety controls, hosting region or tool behaviour without notice. Classify changes by impact. A routine security patch may be automatic; a new model or changed action policy should trigger notice, release notes and your regression set. Require a rollback or containment path for material deterioration.

If the supplier depends on another model or platform, ask what happens when that service changes price, capability, terms or availability. Resilience may mean a fallback model, degraded mode, manual queue or supported migration—not a vague promise of “multi-model architecture.”

Calculate the whole operating cost

Request a worked bill using your volumes, document sizes, call duration, message patterns, connected tools, storage, environments and support tier. Add internal review, exception handling, monitoring, evaluation maintenance and incident time. Ask how the system behaves at a budget limit: alert, slow, degrade or stop.

Compare cost per accepted outcome. Cheap generated output can be expensive when staff verify, correct and re-enter it. A higher unit price may be justified if the evidence, controls and integration reduce hidden operational work.

Make renewal a fresh risk decision

Do not let the contract renew solely because the agent is already embedded. Review actual outcomes, critical incidents, corrections, permission growth, new subprocessors, model changes, cost, portability and remaining business need. Re-run a subset of acceptance cases. Remove unused access and data. If the exit test has never been exercised, test an export before leverage disappears.

The final procurement record should show why this vendor, configuration and authority level were chosen; which risks remain; who accepted them; and what would cause suspension or exit. That decision trail is more valuable than a high aggregate score.

Red flags that should pause the purchase

• The vendor cannot map data, subprocessors or retention.
• The agent requires a shared administrator account.
• Evaluation evidence is a generic benchmark unrelated to the workflow.
• Human escalation exists in slides but not in the product demonstration.
• Logs show messages but not tool arguments, approvals and results.
• Material model or policy changes can occur without notice or regression.
• Export omits workflow configuration, action history or evaluation evidence.
• Deletion and transition duties disappear into “commercially reasonable” language.

One red flag may be resolvable through configuration or contract. Several usually signal that the buyer will own risks the product makes difficult to observe.

Primary sources

• NIST — Cybersecurity Supply Chain Risk Management Due Diligence Guide (SP 1326)
• NIST — AI Risk Management Framework
• NIST AI Resource Center — Govern third-party risk
• ICO — AI contracts and third parties

Continue the implementation path

• Build, buy, or integrate an AI solution
• The true operating cost of an AI agent
• AI automation contract requirements