AI Agent Evaluation: The Working Playbook

AI agent evaluation measures whether an agent completes tasks correctly, safely, and at an acceptable cost, across the whole run rather than only the final message. Agents plan, call tools, and act in the world. A good final answer can hide a wasteful or dangerous path, and a confident "done" can hide a task that never happened.

What to evaluate: outcome, trajectory, cost

Outcome. Did the task actually complete? Check the resulting state (the booking exists, the file changed, the record updated), not the agent's claim. The verification recipe builds claims-versus-evidence checks.

Trajectory. Did it use the right tools in a sensible order, without unnecessary or risky steps? A run that succeeds after deleting and recreating a directory is a warning, not a pass.

Cost and latency per task. Agents can be cheap. One builder's browser agent completed a flight search in about 7 seconds for $0.004, as reported in the build entry. A per-task number like that is the baseline your agent should be compared against.

Judge questions for agents

Closed questions make agent runs scoreable at scale:

• Claims versus evidence: "Does the final page show a confirmation number?"
• Tool choice: "Was a search tool appropriate for this step, given the stated goal?"
• Action risk: "Does the next action delete, pay, send, or publish?"

Each returns a probability, so uncertain judgments route to people instead of counting as passes. Writing judge questions covers the wording. Use a judge from a different model family than the agent to avoid self-preference bias.

Offline and online evaluation

Offline: a task suite with known end states (the agent-shaped version of an LLM evals suite), run in a sandbox, several runs per task because agents vary. Report success rates, not single results.

Online: judge a sample of production runs daily. Failed claims escalate to a person, following the human-in-the-loop rules.

Gates before irreversible actions

The rule on this site holds for agents most of all: no irreversible action on a lone verdict (agent verification). Make the pre-action gate part of the evaluation too. Measure how often it fires and how often it was right to. A gate that never fires isn't proven safe; it may simply never be tested.

Scoring an agent release

Five numbers, compared with the last release: task success rate on the suite, median cost per task, median time per task, gate trigger rate, and human escalation rate. A release that raises success but doubles cost is a tradeoff to decide on deliberately, not a win to announce.

Frequently asked questions

How do you evaluate an AI agent?

Score outcomes against real evidence, review the steps taken, and track cost and latency per task, across repeated runs.

Why not just check the agent's final answer?

Agents can report success without achieving it. Verify the resulting state instead.

How many runs per task should an evaluation use?

Several, because agent runs vary. Report success rates rather than single outcomes.

Can a judge model evaluate an agent?

Yes, for closed questions about evidence and steps. Use a different model family from the agent's. The judge side is covered in what Jev is.

What's the biggest agent evaluation mistake?

Trusting the agent's own report that the task is done.