A rubric-scored evaluation assumes the verdict comes from reasoning about a response against a criterion. This paper tests that assumption twice, and it fails both times. First: train a small text classifier on the rubric wording alone — never showing it the response, the conversation, or anything the model produced — and it predicts the judge's yes/no verdict 83% to 90% of the time. Second: hold the criterion fixed and rewrite the response so it plainly violates what it previously satisfied, and the judge changes its mind on only 38% of pairs. Reverse the criterion instead and it moves on 17% to 32%. The score is partly a property of how the criterion is phrased, and it is stickier than it looks.
The pipeline under examination is the one most teams now run: a conversation, a response to grade, and a criterion written by a human — "the answer should tell the patient to seek urgent care", "the answer should not speculate beyond the cited evidence". A judge model reads all three and returns a binary satisfied or not. The rubrics here are not model-generated filler; the medical ones are clinician-authored and case-specific, which makes them the best case for the method rather than a straw man.
The experiment is a subtraction. Remove two of the three inputs — the conversation and the response — and see how much of the verdict is still predictable from what remains.
The probe is a small domain-pretrained text classifier trained on rubric text and nothing else, with the judge's own outputs as labels, under five-fold cross-validation with bootstrapped confidence intervals. If judging were reasoning over the response, this classifier would have nothing to learn and would sit at chance. Step through the four sets of responses it was trained against.
Raw accuracy is the number that overstates the case, and the paper says so: these label sets are heavily imbalanced, with a majority-class guess alone reaching somewhere between 76% and 89% depending on which model's responses are being graded. Balanced accuracy is the honest reading — it weights both classes equally, so 50% is a coin flip — and it still lands between 80% and 88%. The rubric wording is carrying a large share of the verdict on its own.
The second experiment is a counterfactual and it needs no classifier. Take 500 conversations, one criterion each. Generate two responses per criterion — one written to satisfy it, one written to violate it — and check the contrasts by hand. Then hold the conversation and the criterion fixed and ask the judge to grade both. A judge reasoning about the response should flip. Sort the 493 usable pairs by what it actually did.
The stuck pairs are the ones that matter for anyone reporting a number. If a judge gives the same verdict to a response that satisfies a criterion and to one written to violate it, then on those items the score is not measuring the response at all — and it will not move when your model gets better or worse either.
The third experiment runs the perturbation from the other side. Over 1,000 conversations, hold the conversation and the response completely fixed and rewrite the criterion so its intended meaning is reversed — a rubric that asked for something now asks for its opposite. A judge grading the response against the criterion should now return the opposite verdict on almost every item.
Put the two counterfactuals together and the failure is symmetric: changing what is being graded mostly does not move the verdict, and changing what it is being graded against mostly does not either. Whatever the judge is responding to, it is more stable than both of the things it is supposed to be comparing.
Neither counterfactual needs the paper's apparatus. Both are a sampling job against an eval you already have: rewrite some responses to be plainly wrong, invert some criteria, and count how often the verdict fails to move. Set a sample size and see what the paper's measured rates would predict.
The numbers above are the paper's rates, on its judge, on its rubrics — your own will differ, and the point of running the check is to find out by how much. What is not sample-dependent is the design lesson the authors draw: a rubric-scored benchmark is only measuring responses to the extent that its criteria carry no recoverable prior of their own. If a criterion's phrasing predicts its own verdict, the eval is partly grading itself.