Ask for a number, hedges disappear

Three papers today, about where the confidence you were reading came from. The first opens a small model up and finds that hedging lives in about twenty features and the numeric-confidence path bypasses them entirely. The second replaces a five-agent code reviewer with three and beats it, by giving the third agent a schema that makes agreement expensive. The third stops letting a planner grade its own training targets and lets a verifier pick them, and the model's plan-acceptance rate on Blocksworld jumps from 1 in 80 to 57 in 80.

Today's pick

80% vs 14%

of the same model's answers were falsely certain when confidence was asked as a 1-to-5 score versus a verbal hedge — same 450 controlled questions, mode is the only thing that changed

Different Facets of Verbalised Overconfidence: an Interpretability Study

Ship any product that reads a model's own confidence back to a user or a downstream branch, and this paper is one you want the eng team to read this week. Qwen3-4B was given 450 open-ended questions built from box-placement stories where the evidence was sometimes enough to derive an answer and sometimes not, and was asked in three ways to say when it wasn't sure: hedge in words, refuse with 'I don't know', or return a 1-to-5 confidence score. On the questions with no derivable answer it stated certainty 14% of the time in words, 27% of the time in the abstention setting, and 80% of the time as the score '5' — the numeric channel effectively skipped the hedge. Opening the model up, the authors identify about twenty middle-layer features that carry the 'I'm not sure' signal as a sparse override on top of a broad certainty coalition; suppressing those twenty collapses honest-uncertainty answers from 99% to 0%, and boosting them ×3 rescues 94% of the verbal-mode overconfidence errors — and moves the numeric-mode error rate from 80% to 80.3%. Treat verbalised confidence as a real signal and numeric confidence as a signal about the format prompt, and hedge in words if you need the model to actually route around what it doesn't know.

open the explainer → abstract PDF

87% vs 82%

of LiveCodeBench passed by a three-agent main+reviewer+critic loop versus a five-agent baseline — one model throughout, structured disagreement in the critic step

Adversarial Review: minimal three-agent code review outperforms five-agent baselines

The received wisdom on multi-agent coding is more roles, more reviewers, more meta-agents; scaled up to five it stops paying, and here it is beaten by three when the third agent is given a schema that makes agreement expensive. The reviewer flags bugs on a frozen artifact and the critic must respond with one of AGREE, DISAGREE_EVIDENCE citing code that contradicts the flag, or DISAGREE_CONCERN raising an epistemic worry the reviewer must then answer with code — the reviewer is explicitly barred from capitulating to a bare 'are you sure?'. On identical Claude Sonnet 4.5 runs the three-agent protocol scores 87% on LiveCodeBench versus 82% for the five-agent MARS baseline and 77% for a single-reviewer or self-refine loop, 75.2% on SWE-bench Verified versus 71.6% for a zero-shot agent, and — after a first attempt failed on SWE-PRBench because reviewer and critic collapsed into false consensus — F1 climbs from 0.457 to 0.533 with the same fix. If the workflow you are shipping runs a reviewer against a coder, cite-required disagreement is the lever that turns the extra call into evidence rather than agreement.

open the explainer → abstract PDF

1 → 57

of 80 held-out Blocksworld plans accepted by an external verifier, before and after LoRA-training the base model on verifier-accepted attempts — no oracle, no stronger teacher, 63 seconds on one A100

Governance Records as Supervision for Workflow Repair

The usual story for a self-training loop is that the model grades its own attempts and trains on the ones it liked. Held to a proper external verifier — VAL on PlanBench replanning tasks — Qwen3-14B out of the box gets 1 plan out of 80 accepted; put it into thinking mode and it gets 30, at ~66 seconds per plan; do 72 optimizer steps of LoRA on its own thinking-mode attempts that VAL accepted, and the same non-thinking mode gets 57, at 1.2 seconds per plan. The ablation is where the paper earns its title: on 160 fresh cases, training on the same candidate pool but letting the model pick its own targets gets 69 accepted, letting the verifier pick gets 102, and the pairwise gap is significant at p=2×10⁻⁶ — the training signal that matters is who confirmed the attempt, not who produced it. Where you already run a verifier for safety or format, log every accept as a governance record and use those records as free supervision instead of buying reasoning-model inference for every request.

open the explainer → abstract PDF