Rerankers, reward models and multi-document scorers all do the same thing: put the candidates in one prompt and grade them together. That means every score is a function of the order the candidates arrived in — and the metric you select the scorer on cannot see it. Five trained scorers within 0.010 of each other on ranking quality kept sets that overlapped by only 0.66 to 0.84 when the same candidates were reordered, and a purpose-built commercial reranker reproduced just 0.667 of its own kept set. Nothing available at prompt time removed it: the one intervention that did improve ranking quality left all three downstream decisions exactly where they were. Training the score not to depend on order does work, cutting a reader's answer flips to 0.125 of reordered pairs against 0.149–0.164 for three other objectives that also target order.
The architecture is the cause. To grade twenty candidates, they go into one prompt together and come back with twenty scores. That is efficient and it is also the whole problem: candidate seven's score depends on what candidates one through six looked like, because they shared the context window.
Nobody selects a scorer on that. They select on ranking quality — a metric computed over the ordering the scorer produces, which averages away exactly the instability that matters. But nothing downstream consumes an ordering. A threshold consumes a set. A reader consumes a top-5. A preference model consumes a chosen-and-rejected pair. Those are decisions, and they can flip while the ranking metric does not move.
The measurement is simple enough to copy this week. Take the same candidate set, permute it ten times, score each permutation, and compare what the frozen consumer decides each time. For a threshold, that is the average pairwise overlap between the kept sets. For a reader, it is how often the answer flips across a pair of permutations.
The result is a gap between two columns that should have moved together and did not. Five trained scorers span 0.010 on ranking quality — a rounding error, the kind of difference nobody would ship a change for. On the decision measure they span 0.656 to 0.835 kept-set overlap, which the authors point out is more than an order of magnitude larger.
Three prompt-time interventions were tested, and the pattern across them is the useful part. Round-robin partitioning — distributing consecutive first-stage ranks across separate windows — recovered 0.052 of ranking quality, a genuinely large gain, and left the kept set, the reader's verdict and the preference pair exactly where they were. Post-hoc logit calibration improved neither quality nor overlap. No alternative answer skeleton lowered order sensitivity distinguishably.
That first one deserves the emphasis. It is the clean demonstration that the two things are separable: you can materially improve the metric everyone reports and change none of the decisions anyone acts on.
The working objective adds one term. Anchor each candidate's score to a teacher target as usual, then penalise how much the scorer's own score for that candidate varies across shuffled views of the same window. Two views are enough, and only the first carries supervision.
It is worth separating that from the nearest alternative, because the difference is instructive. Order-averaged distillation precomputes an average over ten shuffled teacher orders and bakes it into the label — it targets invariance through the target. This targets invariance in the student. Permutation augmentation alone, training on the same shuffled views without the penalty, closes about two thirds of the gap; the penalty closes the remaining third.
The payoff shows up where it should: a reader's answer flips on 0.125 of permutation pairs, against 0.149 to 0.164 for the three other objectives that also target order and 0.177 for the plain baseline. It stays below order-averaged distillation on all 12 base models tested, and one of its permutations retains sets that overlap more than ten averaged off-the-shelf permutations.
You do not need to retrain anything to find out whether this applies to you. Take a query, shuffle the candidate list ten times, score each, and look at what your threshold retains each time. The overlap between those sets is a number you can produce this afternoon, and it is far more informative than the ranking metric on your dashboard.
Drag the overlap down to what the paper measured and watch what it means at your volume. This is the count of decisions that would have gone the other way if the same candidates had arrived in a different order.