How to Write LLM Judge Questions (The Skill Behind Every Good Build)

Spend a week reading 600+ builds and one pattern outshines every model spec: the difference between a judge pipeline that works and one that flips coins is almost never the model. It's the questions. Two builders, same model, same task; one asks "is this email urgent?" and gets astrology, the other asks "does this email state that a customer-facing system is currently down?" and gets a classifier. Question design is the actual skill of the decision-model era, and unlike model weights, it's fully learnable. This page is the craft.

The one rule everything else derives from

A good judge question could be answered consistently by a new hire on day one. Not because it's easy, because it's operational: it names observable evidence and a closed decision. If two reasonable people could disagree after reading the same text, the model will disagree with itself across runs, and no amount of prompt seasoning fixes a question that isn't a question.

The transformation always runs the same direction: from a quality ("urgent", "toxic", "good lead") to a test for evidence of that quality. Urgency becomes "requires action within 4 hours to prevent customer harm: yes/no." Toxicity becomes "contains a threat of physical harm against a specific person or group." Lead quality becomes three separate questions about problem-fit, budget language, and authority, because compound qualities are stacks of simple ones wearing one word.

The patterns that keep working

Decompose ruthlessly. One judgment per question. The SuperX scorer asks 61 questions per draft rather than "is this post good?", and at reported per-verdict prices decomposition is free, while each answer becomes independently debuggable.

Close the answer set, and close it honestly. Labels the model must choose among, with an explicit escape hatch: "unclear" or "insufficient evidence" as a legal answer. Deny the escape hatch and the model launders uncertainty into your other categories, silently.

Define the boundary cases inside the question. Most verdict flips live at edges you didn't legislate. "Does the review mention a defect? (Cosmetic damage counts; disliking the color does not.)" Two parentheses, half your ambiguity gone.

Anchor scores or don't use them. A 1-to-10 scale without anchors is vibes with digits. Either define what a 3 and a 7 are, or collapse to the three-way ruling you actually needed: fail / borderline / pass.

Ask about the text, not the author. "Does the message contain X" is checkable; "did the sender intend X" invites the model to write fiction. Judges rule on evidence; keep them off the couch.

The failure modes to test for deliberately

Before trusting any question, probe it: negation traps ("no issues at all!" must not flag as an issue), fluency bias (confident well-formatted wrongness must still fail; see the judge guide's bias list), length sensitivity (the same fact in one line and five paragraphs should get one verdict), and instruction bleed (text that talks about your categories, like an email quoting your spam policy, must be judged for what it is, not what it mentions). Fifty labeled cases per question, checked against your own answers, is the calibration ritual; the evals guide turns that ritual into permanent infrastructure, and getting started walks the first pass.

Then version everything. A reworded question is a new instrument; tag verdicts with the question version or accept that your metrics have quietly become archaeology.

Frequently asked questions

What makes a good LLM judge prompt?

Operational wording (observable evidence, closed answers, an honest "unclear" option), boundary cases legislated in-line, one judgment per question, and calibration against 50+ known cases before production. The model matters less than people hope; the wording matters more.

How many questions should I ask per item?

As many as the task decomposes into; cost stops being the constraint at decision-model prices. Cataloged builds run from one question per email to 61 per draft. Debuggability, not budget, sets the number.

Should judge questions include examples?

Boundary examples inside the question ("X counts; Y does not") earn their tokens; long few-shot galleries usually add latency faster than accuracy for closed-set rulings. Legislate edges, don't anthologize them.

How do I stop verdicts from flipping between runs?

Sharpen the question first (ambiguity is the usual culprit), add the explicit unclear option, pin question wording under version control, and route residual low-confidence cases upward per the cascade pattern instead of pretending they're stable.

Where can I see real question sets?

The build entries across triage and research link authors' write-ups, and the Jev Engineering guide is this site's deeper companion on the craft.