Who Judges the Judge?

The standard way to test an AI feature is to use a second AI to grade the first one. It works well enough that most teams stop thinking about it. But an LLM judge is a measuring instrument nobody calibrated, and an uncalibrated instrument doesn't fail loudly — it returns a number, in the right range, with the right number of decimal places, and you act on it. That's the failure mode worth designing against.

Every scoring rule has a cheapest way to win

When I built evalgate, an open-source pre-merge quality gate for LLM features, the hard part wasn't the scoring. It was noticing how many perfectly reasonable scoring rules quietly reward the wrong behavior.

Grounding is the clearest example. You decompose an answer into atomic factual claims, check each one against the source documents the model was given, and score what survives. Reasonable. Now ask what happens when the judge returns eight verdicts for ten claims.

The obvious implementation scores the eight it got back. That's a bug with a very specific shape: the judge just raised its own score by answering less. Drop the two claims it was least sure about, shrink the denominator, watch grounding go up. The more unreliable the judge, the better the number looks. So a dropped verdict has to count as unsupported. The missing claims stay in the denominator, and a judge that answers less scores worse — which is the direction the incentive has to point.

Every scoring rule has a cheapest way to win. Find it before your judge does.

The same pattern shows up in retrieval. Precision asks what fraction of the retrieved documents were relevant. If the retriever returns nothing at all, the honest reading is that no irrelevant documents were returned — precision 1.0, a perfect score. Which is how a completely dead retriever reports as flawless. So empty retrieval scores zero. Vacuous truth is still truth, and it will still ship to production.

A gap and a lie are different bugs

Most faithfulness metrics collapse every unsupported claim into one bucket. But there's a real difference between a model saying something the sources don't mention and a model saying something the sources directly contradict.

The first is a gap. The second is a lie. If your customer service agent omits the return window, that's incomplete. If it says shipping is free over $50 when your policy says $75, you have a different problem — one with a dollar amount attached, and eventually a chargeback.

So contradiction is tracked separately from absence, and a single contradiction can fail a case outright while ten omissions are tolerated. That's not a scoring nuance. It's the difference between a metric and a decision.

Calibrating the judge

Which brings back the circular problem. If a model is grading your model, what grades the grader?

The answer is unglamorous: the judge gets its own test suite. A set of cases with human scores, committed to the repo. Run the judge against them and you get three numbers — agreement, bias, and correlation. Then hold the output constant and vary only the judge, because that's the only way a score change is attributable to the judge rather than to your application.

Splitting bias out from agreement matters more than it sounds. A judge that's uniformly 0.2 generous is fixable by moving a threshold. A judge that's 0.2 off in random directions is not fixable at all. Averaged into one score, those two very different situations look identical.

And the calibration set itself needs a guard. If every case in it should score 1.0, then a judge that returns 1.0 unconditionally calibrates perfectly. So a set with no spread is rejected before it runs — it has to contain at least one case the judge is supposed to fail. Correlation gets the same treatment: when the human scores don't vary, it reports null rather than 0, because the question was unanswerable and 0 would read as disagreement.

Then the agreement number rides along with every score that judge produces, bound to that judge's ID. Swap the judge and it can't inherit the old one's credibility. If judge–human agreement is 0.71, every downstream score carries that 0.71 with it, where a reader can see it.

What this is really about

Every one of these decisions is the same decision wearing different clothes: find the path where the system reports success without earning it, and close it.

That instinct doesn't come from machine learning. It comes from twelve years of program delivery, where the status report is green until the week before the deadline. You learn to distrust any metric that can't explain itself, and to ask what a number would look like if the underlying thing were quietly broken. Most of the time the answer is: exactly the same.

AI systems make that failure mode cheaper and faster. They produce confident, well-formatted output regardless of whether anything underneath is working. The output of a broken RAG pipeline and a working one look identical until you read the sources — which nobody does at scale, which is the whole reason to automate the reading.

The teams that will get real leverage out of AI over the next few years aren't the ones with the best prompts. They're the ones who can tell, on any given Tuesday, whether the thing still works — and defend the answer when someone asks how they know.

This is the engineering discipline underneath every AI system we build at HCT, for clients across Charlotte and Lake Norman who need something more durable than a demo. See how we approach AI engineering →