An LLM judge is a classifier, so score it before you trust it

The idea

Using a model to grade outputs is how most teams scale evaluation past what humans can read. The catch is that the judge is itself a binary classifier. It says pass or fail, and it is wrong some of the time in both directions.

That matters because the number you report, the pass rate, is a blend of real quality and judge error. If the judge approves bad answers often, a system that produces more bad answers can still score well. You are reading the world through a noisy instrument without knowing how noisy it is.

The fix is to measure the instrument. Label a sample of real outputs by hand, run the judge on the same sample, and compute two rates: how often it passes good outputs (true positive rate) and how often it fails bad ones (true negative rate). With those, a judge score becomes interpretable. You can even correct it: observed pass rate equals TPR times the true rate plus the false positive rate times the remainder, which you can solve for the true rate.

An example

Take a support assistant with a grounding check. The judge prompt asks whether every claim in the answer is supported by the retrieved passages.

• Sample a couple of hundred production answers and label each one by hand as grounded or not.
• Run the judge on the same set and build the confusion matrix.
• The judge agrees with humans on obvious cases, but it passes most answers that are fluent and cite a passage that does not actually contain the claim. Its true negative rate is poor.
• A prompt change that makes answers sound more confident now raises the judge score, while human review finds more unsupported claims.

Without the labeled set, that change ships as an improvement. With it, the fix is clear: rewrite the judge to split the answer into atomic claims and quote the supporting span for each, then measure again.

Where it breaks

If your human labelers disagree with each other, there is no ground truth to validate against. Measure agreement between annotators first. A rubric humans cannot apply consistently is a rubric problem, not a judge problem.

Error rates are not fixed. A judge validated on last quarter's outputs may behave differently once a new model writes in a different style. Judges also carry known biases: favoring longer answers, favoring the first option in pairwise comparisons, and favoring outputs from their own model family.

The correction formula assumes error rates are constant across the data. They often vary by slice, so one global correction can hide a slice where the judge is nearly useless.

And where a deterministic check exists, such as schema validation, exact match or a unit test, use code. A judge there only adds noise.

In practice

• Label before you write the judge prompt, so the rubric is shaped by real failures rather than imagined ones.
• Prefer binary, single-question rubrics over a numeric scale. They are easier to label and easier to measure.
• Report true positive and true negative rates separately. Accuracy hides a lenient judge when most outputs are good.
• Keep a held-out labeled set. Do not tune the judge prompt on the examples you use to measure it.
• Pin the judge model version and temperature, and re-validate whenever the judge, its prompt or the system under test changes.
• For pairwise judging, run both orderings and count disagreement as a tie.
• Route judge and human disagreements from production into a standing labeling queue so the validation set keeps up with the system.

Treat an LLM judge as a classifier with its own error rates, and do not trust its score until you have measured those rates against human labels on the outputs it will actually grade.