RAG Evaluation: Measuring Whether Your Retrieval Actually Grounds Anything

RAG (retrieval-augmented generation) was sold as the cure for hallucination: ground the model in your documents and it can only say true things. Everyone who's shipped one knows the fine print: RAG doesn't prevent ungrounded answers, it makes them checkable, and almost nobody checks. The retriever fetches something plausible, the model writes something fluent, the demo works, and six weeks later a customer quotes your bot citing a policy that doesn't exist. RAG evaluation is the discipline of checking, and it just became cheap enough to do continuously instead of ceremonially.

Custody note: the evals guide covers building suites in general; this page owns the RAG-specific metrics, which decompose beautifully into judge verdicts because a RAG answer has exactly three ways to fail.

The three failure surfaces, three question families

Retrieval relevance: did we fetch the right material? Per retrieved chunk: does this passage contain information that addresses the query? A stack of per-chunk yes/no verdicts gives you precision (how much fetched junk) and, against a labeled set, recall (what needed material got missed). Most "the model hallucinated" tickets are actually retrieval misses in costume; measure here first and you'll reassign half your bug reports.

Faithfulness: is every claim in the answer supported? The crown-jewel check. Decompose the generated answer into claims (yes, that split is itself judgable: "does this sentence assert a checkable fact?") and ask per claim: is this supported by the provided passages, contradicted, or absent? "Absent" is the hallucination detector; "contradicted" is worse and rarer and should page someone. This exact verdict, run in production per response, is the grounding guardrail from the guardrails page; run offline across a test set, it's your faithfulness score. Same question, two speeds.

Completeness and answerability: should we have answered at all? Does the answer address what was asked; do the retrieved passages actually contain enough to answer; and, the underrated one, when the corpus doesn't contain the answer, did the system say so instead of improvising? Refusal-when-ungrounded is a feature you have to measure or you won't have it.

Running it at decision-model prices

The arithmetic that changes behavior: a 200-query eval set, five chunks each, answer decomposed into a handful of claims, lands around a few thousand verdicts per full run, which at reported pricing (the reference: ~26,000 verdicts for $0.13) costs cents. So you run it on every retriever tweak, every chunking change, every embedding swap, every prompt edit, per-commit like the evals discipline prescribes, instead of per-quarter like the consulting deck prescribed. Standing rules apply: operational question wording, cross-family judging (a decision model grading your chat model's faithfulness is cleaner than self-grading), versioned questions, and a human-audited sample so the instrument stays calibrated per labeling discipline.

One trap worth naming: aggregate scores hide the interesting failures. Keep per-question, per-chunk verdicts queryable (materialize them) so "faithfulness dropped 4 points" decomposes into which claims, from which chunk types, on which query classes, in one query instead of one afternoon.

Frequently asked questions

What is RAG evaluation?

Measuring a retrieval-augmented system on its three failure surfaces: retrieval relevance, answer faithfulness to retrieved material, and completeness/answerability, typically via automated judge verdicts against a test set.

What's the most important RAG metric?

Faithfulness, because unsupported claims are the failure users can't detect themselves; but debug retrieval first, since fetch misses masquerade as hallucinations and are cheaper to fix.

Can RAG evaluation run in production, not just offline?

Yes: the faithfulness verdict doubles as a per-response grounding guardrail at milliseconds and fractions of a cent, with failed checks routed to regenerate or escalate. Offline suites and live checks should share question wording.

How big should a RAG eval set be?

A few hundred queries drawn from real traffic, including known-unanswerable ones to test refusal, growing with every production failure. Twelve hand-picked demo queries produce optimism, not measurement.