LLM Evals: How to Build an Eval Suite That Actually Catches Regressions

Every AI product team learns the same lesson on the same schedule: around month two, a prompt tweak that "obviously improved things" quietly breaks a case that mattered, a customer finds it first, and suddenly everyone believes in LLM evals. This page is for skipping to the believing part.

An eval suite is to an AI product what a test suite is to software: a fixed set of inputs, a definition of good output, and an automated grader that tells you whether today's system beats yesterday's. The grader is where teams stall, and where the economics just changed. (This page covers building the suite; the grading pattern itself is the LLM-as-a-judge guide, and question craft lives in how to write judge questions.)

The three layers of a working eval suite

Layer 1: assertions. Plain code checks: did it return valid JSON, is the required field present, is it under the length cap. Free, instant, catches the embarrassing 20 percent. Never skip these just because AI graders exist.

Layer 2: judge verdicts. The heart of the suite. For each test case, structured questions graded by a model: "Is the answer supported by the provided source?" "Does the reply follow the refund policy?" "Is the tone appropriate for a complaint?" This is where a decision model like Jev changes the operating rhythm. One cataloged builder ran roughly 26,000 verdicts for $0.13 (build, numbers as reported); at that price, a thousand-case suite with ten questions each costs less than the coffee you drink while it runs, so you run it on every commit instead of every release. Catching regressions per-commit versus per-release is the entire game. The ecosystem already publishes shared jev evals; write yours the same runnable way.

Layer 3: human spot checks. A rotating sample of judge verdicts reviewed by a person, because your grader is a measurement instrument and instruments drift. Ten minutes a week keeps the whole tower honest.

What to actually put in the suite

Golden cases (your product's must-never-break moments), regression cases (every bug that ever shipped becomes a permanent test), adversarial cases (prompt injection attempts, hostile users, nonsense input), and boring cases (the median query, because optimizing for edge cases quietly degrades the middle). Aim for a few hundred cases before trusting the numbers; a twelve-case suite produces vibes with decimal points.

Two rules that save quarters of pain. First, version your judge like an API: a change to a judge question is a breaking change to every historical score, so tag suite results with the judge version and never compare across it. Second, judge with a different model family than the one being judged; self-preference bias is real and flattering.

The escalation pattern for hard grades

Some grades are genuinely subtle: nuanced tone, complex policy edge cases. Don't push those onto the cheap judge and don't pay frontier prices for everything either. Copy the cascade from the fraud build: fast judge grades everything, low-confidence grades escalate to a frontier model or the human sample. Reported result there was 96/100 accuracy at ~$0.07 per hundred items, and eval grading is a friendlier task than fraud.

Frequently asked questions

What are LLM evals?

Automated test suites for AI systems: fixed inputs, defined success criteria, and graders (code assertions plus judge models) that score outputs so you can compare versions objectively instead of by anecdote.

How many eval cases do I need?

Enough that a single flipped case doesn't move your headline number: a few hundred is a working floor, seeded from real logged traffic plus every bug you've ever shipped. Grow it forever; never delete a regression case.

Are LLM judges reliable enough to grade evals?

With narrow questions, cross-family judging, versioned prompts, and weekly human spot checks, yes; it's the industry-standard method. The known biases and their fixes are covered in the judge guide.

What does it cost to run evals on every commit?

At builder-reported decision-model prices (tens of thousands of verdicts for cents), a serious suite runs for well under a dollar a day. The cost table lives in what builds actually cost.

Evals vs monitoring: which first?

Evals, narrowly: you need a definition of good before production telemetry means anything. Then feed real production failures back into the suite so the two systems sharpen each other.