How to Evaluate an AI Sales Agent: A 12-Point Business Checklist
A fluent answer is not enough. Test account identity, evidence, access, commercial claims, approval, idempotency, read-back, and recovery.
The final answer is only one part of the test
OpenAI's agent workflow evaluation guide points to traces of tool calls, handoffs, and guardrails. Its evaluation best practices emphasize tool arguments, edge cases, and repeatable tests. Anthropic's security-practice discussion also illustrates why execution environment and configuration matter to agent behavior.
The 12 checks below are a KnowSales editorial framework for sales workflows, not a vendor standard. They cover the path from reading customer context to making a durable change. They are particularly useful when an agent connects CRM, product knowledge, and customer communications.
Six checks before the agent acts
| # | Gate | What a passing run shows |
|---|---|---|
| 1 | Account identity | Company name plus an independent hard identifier; ambiguous matches remain candidates |
| 2 | Access scope | The agent can read only customers, knowledge, and files allowed by the current user and tool grant |
| 3 | Openable evidence | Important claims link to the underlying email, document, record, or page |
| 4 | Time validity | Expired quotes, old delivery estimates, and former owners are treated as history |
| 5 | Claim status | Buyer request, company confirmation, third-party statement, and open question stay distinct |
| 6 | Account isolation | Account A's prices, history, and contacts never appear in Account B's answer |
Test these with deliberately similar synthetic accounts. A system that works on obvious examples has not proved it can avoid account crossover. Include similar company names, buyers in the same country, and two customers interested in the same product.
Six checks when the agent acts
| # | Gate | What a passing run shows |
|---|---|---|
| 7 | Write plan | Target object, existing value, proposed value, source, and impact appear before the write |
| 8 | Human approval | Customer facts, price commitments, and external content obtain approval when required |
| 9 | Correct record type | Stable company facts go to profile; one-off interactions to activity; reusable knowledge excludes account-private details |
| 10 | Idempotency | Retry, timeout, or refresh cannot create an identical second record |
| 11 | Read-back | The agent fetches the actual record ID and checks account, text, business time, and permission state |
| 12 | Recovery | A wrong write has a recoverable path; failure preserves record ID and evidence for diagnosis |
For more detail on claim status and record type, see the CRM write-plan guide and sales agent governance checklist.
Four synthetic scenarios to start with
- Two similar customer names interested in the same product test identity and isolation.
- An expired quote alongside a newer activity tests temporal reasoning.
- āWe might buy if the trial passesā tests conditional language and claim status.
- A write succeeds but the response times out, testing idempotency, read-back, and duplicate prevention.
For each scenario, keep the input, expected entity, permitted actions, actual tool trace, and persisted result. A single successful run proves only that case. Repeat with different user roles, languages, and client entry points before expanding a high-risk workflow.
Make it an acceptance standard
Ask a vendor or internal team to demonstrate these scenarios instead of only a polished final answer. Decide which failures are unacceptable: cross-account leakage, authorization bypass, and duplicate writes often deserve zero tolerance. Decide which can be mitigated by review during an early rollout, such as phrasing or answer order. Grade failures by impact and rerun the original scenario after a fix.
KnowSales has observable product surfaces across customers, knowledge, MCP, permissions, citations, and dashboards. Whether a particular tool is available to an account still depends on current permission and live behavior. A limited rollout such as durable tasks needs separate acceptance; this checklist does not confer general availability.
FAQ
Why inspect the tool trace if the answer is correct?
An agent can reach a correct sentence through the wrong source or an unauthorized call. The trace shows how it arrived there.
Must all 12 checks be automated on day one?
No. Start with the four synthetic scenarios and manually review the high-risk gates, then automate stable regression cases.
Can an occasionally failing agent be released?
That depends on the failure. Wording can receive human review. Cross-account exposure, unauthorized access, and duplicate writes cannot be dismissed as occasional.
Why is read-back stronger than a success response?
A tool response does not prove the content landed in the right account, record type, and business date. Fetching the persisted record does.
Use the checklist on a read-only account brief first, then explore KnowSales connections before adding reviewed writes.