Add Verification to Any Agent in an Hour

Agents are confident narrators. "Done! I've updated the config and all tests pass" is a sentence an agent will happily produce whether or not any of it happened. The fix is a checker that reads the evidence and answers narrow questions about it. This recipe shows how to verify AI agent output with Jev, TypeSafe AI's decision model, bolted onto an agent you already run.

Why verification works, the "claims become questions" pattern, and the design rules behind it live in AI agent verification. This page is the one-hour setup.

Builds that already do this

One builder added Jev to a coding-agent loop with a single job, answering "is the task done?", and describes the result as a "wait, what" moment (build). jev-spec checks generated code against the specification that asked for it (build). Another write-up guards a Google ADK agent whose output was valid JSON with the wrong content (build), which is the failure schema validation can't see.

None of these publish accuracy numbers, and there are no official Jev benchmarks. The pattern is consistent across them, which is what this recipe copies.

The one-hour recipe

1. List what the agent claims (10 minutes). Pick one task type your agent performs often: filling a form, updating a record, writing a summary, fixing a bug. Write down the claims it makes when it finishes. Each claim is a future question.

2. Collect evidence, not the agent's word (15 minutes). Verification judges artifacts: the diff, the API response, the final page state, the record after the update, the test output. Wire your agent runner to capture these at the end of each task. If you only have the agent's self-report, you can't verify anything, you can only re-read its optimism.

3. Write the verification questions (15 minutes). Pair each claim with evidence:

   • "Given this task description and this diff, does the diff implement the requested change? YES / NO / UNCLEAR"
   • "Does this final form state contain a value for every required field listed in the task? YES / NO / UNCLEAR"
   • "Does this summary state anything not supported by the source document? YES / NO / UNCLEAR"

   Narrow beats broad. "Is this correct?" is a coin flip; "Does the record's email field match the email in the request?" is a verdict.

4. Add the check step (15 minutes).

# pseudocode, not real API syntax
result   = agent.run(task)
evidence = collect_evidence(result)
checks   = [jev.choose(q, pack(task, evidence), ["YES", "NO", "UNCLEAR"]) for q in VERIFY_QUESTIONS]

if all_pass_confidently(checks):
    mark_verified(task)
elif any_confident_fail(checks):
    retry_or_escalate(task, checks)      # cap retries
else:
    send_to_human(task, evidence, checks)

Per ecosystem documentation, the model is typesafe-ai/jev via the Vercel AI Gateway, returning per-choice probabilities. Real syntax at docs.typesafe.ai.

5. Run it on 20 past tasks (5 minutes plus reading). Replay recent agent runs where you know the right answer. Look at where the checker disagrees with you. Adjust question wording first, thresholds second.

Where verification sits relative to action

Verification after the fact is useful. Verification before an irreversible step is essential. If your agent is about to send an email, delete data, spend money, or deploy, put the check before that step, and even then don't let a single passing verdict authorize it. Use the verdict to decide whether to ask a human, not whether to skip asking.

One builder tested Jev as a pre-action safety monitor that checks each agent action before it runs, and reports it caught most attacks with almost no false blocks while being much faster than Gemini (build). That's reported, not benchmarked, but it's the right place in the loop.

Agent claim checking without new failure modes

The obvious objection: "Isn't this just a model grading a model?" Yes, which is why the checker gets evidence rather than the agent's narrative, and asks narrow closed questions rather than "rate this work". A decision model returning YES/NO/UNCLEAR with a probability is a very different animal from a second chat agent writing a review.

Cap the retry loop. A failed check that triggers a retry that triggers a check can spin forever. Two retries, then a human.

Log every verdict. Task ID, questions, verdicts, confidences, and what happened next. When something slips through, the log tells you which question needs work.

Checks complement output guardrails. Verification asks "did the agent do the task?"; LLM guardrails ask "is this output safe to show?" Many systems need both.

Frequently asked questions

How do I verify AI agent output without re-running the task?

Capture evidence at the end of each run (diffs, API responses, final state) and ask a checker narrow yes/no questions about it. The pattern is laid out in AI agent verification.

Can a verification check approve irreversible actions?

No. Place the check before risky steps, but use it to decide when to ask a human, not to skip asking. Sends, deletes, payments, and deploys need more than one verdict.

Does verification slow the agent down?

Jev verdicts are designed to be fast, and builders report sub-second decisions in several workloads, as reported. The check usually costs far less time than the task it verifies.

What if my agent only reports text, not evidence?

Then add evidence capture first, since judging an agent's self-report only checks its confidence. The getting-started guide covers breaking a task into verifiable decisions.