Tell it exactly what broke.

Three papers about the text you wrap around the model rather than the model itself: the feedback you send after a failure, the way you specify the task, and the rubric your judge scores against. In each case the wrapper turns out to be doing more of the work — or more of the damage — than the choice of model.

Today's pick

90% vs 27%

tasks repaired within four turns — concrete failing input, versus a generic retry

Counterexamples as Feedback for Agent Self-Correction

When an agent's output is wrong, most retry loops send back some version of "that didn't work, try again". This paper builds the alternative and measures it: a deterministic checker runs the candidate against a small held-back test set and returns up to two concrete strings it wrongly accepted and two it wrongly rejected, so the next turn starts from evidence rather than disapproval. On 30 plain-English-to-pattern tasks that repaired 90% within four turns, against 27% for generic self-correction, 23% for a bare failure count and 17% for one zero-shot attempt — and 27 of the 30 had landed by turn three. If your loop already knows why a check failed — the assertion's actual value, the payload the validator rejected, the row that broke — put that object in the retry prompt instead of the word "incorrect".

open the explainer → abstract PDF

158 vs 7,022

tokens to specify one task as a rule, versus as the examples that cover it

LLMs Learn Better In-Context from Rules than from Examples

Adding more few-shot examples is the reflex when a model gets a task wrong. Across five synthetic tasks and fourteen open-weight models, this paper compares stating the rule in words against showing worked examples chosen to cover the same ground, and rules win overall — while adding examples on top of a rule produced no statistically significant gain on any task, and simply scaling the example count gave mostly flat, diminishing or negative returns. The cost gap is severe wherever the task is combinatorial: specifying one arithmetic task took about 158 tokens as a rule and about 7,022 tokens as the demonstrations needed to cover it. The rule advantage is largest when the task is algebraic — same-or-different comparisons, applying an unfamiliar operator — and smallest when it leans on distributional feel or on knowledge the model already carries, which is a usable heuristic for where your context budget should go.

open the explainer → abstract PDF code

83-90%

accuracy predicting a judge's verdict from the rubric wording alone, answer unseen

Judging LLM-as-a-Judge: Concerning Rubric Artifacts in LLM-based Automated Text Generation Evaluation

If you score model output with a judge model against a rubric, this is a reason to audit it. A small text classifier trained only on the rubric wording — never shown the answer being graded — predicts the judge's yes/no verdict with 83% to 90% accuracy on clinician-written medical rubrics, so a large share of the score is recoverable from how a criterion is phrased rather than from what the model wrote. The counterfactual half is worse: rewrite a response so it clearly violates a criterion it previously satisfied and the judge flips as expected only 38% of the time, and reverse the criterion itself while holding the answer fixed and it moves on just 17% to 32% of items. Run both checks on your own eval before you trust its numbers — grade a deliberately wrong answer, and invert a criterion — and see whether the score actually moves.

open the explainer → abstract PDF