Judging pipelines love a two-step: have the model pull out the criteria and the evidence in one call, then reach a verdict from that record in the next. It looks more auditable, more modular, and harder to bias. Across 24,000 judgments on three preference datasets with two frontier judges, it is worse at all three. Agreement with human preferences falls 4–6 points, and the rate at which the judge flips its answer when you swap which candidate came first roughly doubles. Once the record is the only input, every omission inside it becomes part of the evaluator.
The four protocols differ in exactly one dimension that matters: what the model can see at the instant it commits to a verdict. Everything else — the criteria extraction, the evidence quoting, the structured output — is held roughly constant. Step through them and watch what is on the table when the decision gets made.
The motivation for locking is genuinely reasonable, which is why it is worth testing rather than dismissing. For a reasoning-capable model you cannot see the order in which it actually decided things — the visible field order in its output tells you nothing. Persisting the evidence and making it the sole input is an observable alternative: whatever the model was doing internally, you can now point at the artifact and say the verdict came from that.
Two frontier judges, three datasets, 24,000 judgments. The comparison to hold in mind is not locking against nothing — it is locking against the structured one-call version, which does all the same criteria and evidence work but never loses sight of the answers. That is the honest control, and locking loses to it on both measures at once. Switch what the bars show.
Note the direction of the second measure. Order inconsistency is how often the judge changes its winner purely because you showed the answers in the other sequence — pure noise, nothing to do with quality. Structured one-call judging holds it under 10% for both models. Locking pushes it to 16–19%. You did not just lose accuracy, you bought yourself a judge that disagrees with itself one time in five.
The authors are careful about what this is and is not evidence for. It is not a claim about hidden reasoning or internal decision order — those stay unobservable. It is a claim about access: once the deciding call cannot reach the source, whatever the record left out is gone for good. Drag the slider to squeeze the record and watch what falls out of it.
Three specific things a natural-language record does to the thing it summarises, all named in the paper: it can omit a distinction the two answers actually differed on, compress a trade-off into a flat statement that hides which side won it, and preserve position-dependent framing — writing about the first answer as the default and the second as the alternative. The final call cannot repair any of them, because it has nothing to check them against.
This is not an argument against logging what your judge saw, and the paper is explicit about that. Auditability, review, replay and modularity are all real reasons to write the record, and none of them require the record to be the only thing on the table at decision time. The distinction is one line of plumbing. Compare the two wirings.
The principle the authors land on is worth writing on the wall next to your eval harness: the record should assist the decision rather than replace the original pair. And if you want the artifact to be more than decoration, let a later stage correct it when it turns out to be incomplete — a record that can never be revised is a record whose errors are permanent.
An eval suite is a measuring instrument, and the cost of a worse judge is not abstract — it is decisions you make about your own product from numbers that are wrong. Set your suite size and see what the two effects look like as counts rather than percentages.
The cheapest check, and one you can run this afternoon regardless of protocol: take a sample of your judge's comparisons, run each one again with the two candidates swapped, and count how often the winner changes. That number is free, needs no human labels, and if it is anywhere near a fifth then your eval is not measuring what you think it is.