When the grader is the bug

Today's papers converge on one uncomfortable idea: the signals you grade with are shakier than the models you're grading. OpenAI spent last week retracting its own recommendation of a popular coding benchmark after finding roughly a third of its tasks broken. Today's pick measures the same failure from the other side.

Today's pick

32.4%

of a leading coding benchmark's pass/fail verdicts were disputed by an independent judge — against 1.4% for graders written from the task description

DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Tasks

Coding benchmarks grade an agent with the tests that shipped alongside the original human fix — written to confirm that one patch, not to grade an arbitrary one. Auditing 789 runs on SWE-Bench Pro, an independent judge disputed a third of the verdicts: 24% were working patches marked failed, 8.5% were stubs marked passed. Graders rewritten from the task description, asserting only on public behaviour so any working implementation passes, cut that to 1.4%. If your eval replays tests that shipped with a specific fix, much of your model-selection signal is noise.

open the explainer → abstract PDF code

0.75

the share of its correct answers a model could still deliver while staying inside both error budgets — gating on a single confidence score managed 0.31

Two Axes of LLM Abstention: Answer Correctness and Question Answerability

Refusal is usually gated on one confidence score, which conflates two questions: am I likely to be wrong, and is this question answerable at all? A false-premise question — when did Einstein win his second Nobel? — isn't low-confidence, it's unanswerable, and models sit near chance at spotting those however large they get. Telling the model to check the premise backfires: it then disputes sound and false premises alike, and 57% of its challenges are false alarms. A probe on its internal states catches them, and gating on both axes beat any single threshold.

abstract PDF

86.5%

on a terminal-task benchmark from a verifier that needs no training — plus 78.2% on a software-issue benchmark, both the best published

LLM-as-a-Verifier: A General-Purpose Verification Framework

Asking a model to grade something means asking for a score, and it answers with a single token — "7", or "pass" — throwing away everything it knew about how close the call was. This framework reads the whole distribution over score tokens and takes its expected value, giving a continuous, calibrated score with no training, sharpened further by finer scales, repeated sampling, and criteria split into parts. If you rank best-of-N agent outputs with a model judge, this changes how you read the judge, not which judge you buy.

abstract PDF code

When the Judge Changes, So Does the Measurement: Auditing LLM-as-Judge Reliability

A better judge feels like a free upgrade. This audit says a judge swap is a change of instrument: scores move even when the responses being judged are held fixed, and across four datasets only one size step gave a reliable gain — scaling further, or taking a vendor's next API release, did not. Stronger judges reduce but never remove the bias toward longer answers, and sampling one judge repeatedly buys little, because its mistakes are correlated. Pin the version; re-baseline when you bump it.

abstract PDF

Resample or Reroute? Budget-Aware Test-Time Model Selection for Large Language Models

You have a per-query budget and a first answer you don't trust: sample the same model again, or pay more for a stronger one? This paper poses that as online allocation — spend the next unit of budget wherever it buys the most expected correctness — and across an eleven-model open-weight pool it beats fixed routing, one-shot commitment, budget-aware best-of-K, and cascades. The caveat is load-bearing: gains are gated by your verifier and shrink as its quality drops. Get the verifier honest before you build a router on top of it.

abstract PDF code

41.14%

of repository-level functions generated correctly first try, by adding a retrieval signal most code assistants don't have

ProjAgent: Procedural Similarity Retrieval for Repository-Level Code Generation

Code retrieval for a repo assistant ranks by lexical, structural, or semantic similarity, so it misses a function that already solves your problem because it uses different names in a different corner of the codebase. This paper adds procedural similarity: break the target function into reasoning steps, retrieve repo functions that behave the same way step by step, and union that with ordinary semantic retrieval. With a conservative compiler-and-static-analysis repair loop on the output, that beat existing retrieval baselines — a second retriever to stack, not a replacement for the one you have.

abstract PDF