Almost every agent loop now asks the model to look back over its own answer and fix it. Run that second pass under a controlled protocol — same task, same conditions, humans and models side by side — and measure how much closer to the truth it lands. On questions with a fixed answer, model revision gains about nothing: statistically indistinguishable from just sampling again. On judgment calls it goes backwards, costing one model 29.2%. Humans gained in both settings, +6.0% and +17.8%. And when models revise human answers they degrade those too, which locates the fault in the revision step itself.
The reflection prompt is now furniture: review your answer, identify any errors, and revise. It produces text that reads exactly like careful reconsideration, which is why nobody checks it. The protocol here is simple enough to be uncomfortable — run pass one, feed the answer back, run pass two, and score how much the answer moved toward the truth. Then run the identical thing on humans as a control. Step through what comes back.
The human column is what makes this readable. If both humans and models were flat, you would conclude the tasks left no room to improve. Humans gain on all three, so the room exists and the models are not using it. On the judgment task the models do not merely fail to use it — they spend the second pass moving away from the answer.
Reflection does not fail uniformly, and the distinction matters for where you can still afford to use it. On objective tasks the second pass is neutral: the answer moves, but not toward the target, which makes it indistinguishable from re-sampling at the same temperature. On subjective tasks the second pass is harmful: it moves predictions systematically away from the target, and all five models show a statistically significant increase in error. Switch the task and watch the whole column change sign.
Read the subjective column carefully before you ship another self-critique step on a rating, a ranking or a rubric score. Claude-3.5 loses 29.2%. Every model is negative. Whatever the second pass is doing on open judgment, it is not error correction — the paper's account is that it pulls the answer toward the model's own prior, which is exactly the wrong direction when the target is somebody else's judgment.
Here is the structural claim, and it is the part worth carrying around. Revision reduces error only if the second pass carries information the first pass did not have. A model conditioning on its own output has no such information — the output is a function of what it already knew. So the ceiling is not a capability problem to be solved by a better model; it is a property of the loop's shape. Choose what the second pass is allowed to see and watch which side of the line it lands on.
The cross-agent grid is the clean test. Humans revising anything — a weak model's answer, a strong model's answer, another human's — come out non-negative every time. Models revising other models are almost universally negative on the judgment task, despite those inputs being good. The one case where a model reliably improves an input is when the input is worse than its own prior, and even that is not correction: it is the answer being dragged toward where the model would have started anyway.
Reflection has two sub-steps — noticing an error, then repairing it — and the paper separates them. Noticing works better than you would expect on objective tasks: models revise answers they got wrong 21 to 33 times more often than answers they got right. That is a strong detection signal, and it makes the flat gain more damning, not less, because the model is aiming at the right questions and still not landing. Then hand it a perfect error signal and see how much of the gap is repair.
The gold line on the repair view is the random-reshuffle baseline at 33.3% — what you would recover by picking a different option at random once told the answer is wrong. Two of five models sit below it. For those, an oracle telling them exactly which answers are wrong is worth less than a coin flip among the remaining choices, and their real benefit from the oracle is retention: not breaking the answers they already had right.
Most self-critique steps in production are a second full generation: roughly double the output tokens and double the latency on that hop. Set your volume and see the bill, then flip the switch to the version that has something the first pass did not — a retrieval call, a tool result, a different model family. The paper does not price your stack, but it does tell you which side of the switch has a mechanism behind it.
Three concrete moves fall out of this. Keep reflection where the second pass genuinely receives something new — a test result, a retrieved document, a different model's opinion — and delete it everywhere else. Never put a self-critique step on a rating, ranking or rubric score, which is where the sign flips negative. And if you want the detection signal without the broken repair, use the second pass only to flag low-confidence answers for a different resolver, because noticing is the sub-step that actually works.