Almost every agent retry loop returns a verdict: the check failed, try again. This paper builds the other kind of feedback and measures the gap. A deterministic checker runs the agent's candidate against a small held-back test set and hands back up to two concrete inputs it wrongly accepted and two it wrongly rejected, so the next turn starts from evidence instead of disapproval. On 30 plain-English-to-pattern tasks that repairs 90% within four turns, against 27% for a generic self-correction prompt and 23% for a bare failure count — from the same model, on the same tasks, with the same number of attempts. The interesting part is what it costs: nothing but the failing input you already had.
The setup is deliberately small so the feedback is the only variable. A task is a sentence in English — "a string that starts with a digit and ends with a letter" — paired with a reference pattern the reader never sees. The agent proposes a pattern. A deterministic checker runs it against a task-local test set of twelve strings that should match and twelve that should not, under full-string semantics, and produces a pass vector. Nothing is judged by a model.
Failures come back in two directions, and the direction is the whole point. A string the pattern should have matched but didn't means the pattern is too narrow. A string it matched but shouldn't have means it is too broad. The checker returns up to two of each, preferring the shortest witnesses it can find, because a short string isolates the mistake.
Both prompts are the same length in spirit and cost the same to send. One of them contains the answer to the question the model is actually facing — which way am I wrong? — and the other does not.
Every condition gets the same model and the same four turns. Zero-shot is the control: one attempt, no second chance. Generic self-correction is what most harnesses do today — the model is told the answer was wrong and asked to reconsider. Error-only feedback adds the failure count without saying which cases failed. Diagnostic feedback adds the witnesses.
Read the middle two together. Generic self-correction spends 2.77 attempts per task to reach 27%; error-only feedback spends 2.83 to reach 23%. Telling the model how many cases failed, without telling it which, buys nothing over telling it nothing — the number is not information the model can act on. Diagnostic feedback spends fewer attempts than either, 2.53, and lands 27 of the 30 tasks.
A turn budget is a real cost — every extra turn is another model call, another few seconds, another chance to wander. So the useful question is not whether the loop converges but how quickly, and where the curve flattens. Slide the budget and watch what a longer loop actually buys.
Five tasks fall on the first attempt, before any feedback exists — that is the zero-shot rate, and it is the same five for every condition. Turns two and three carry nineteen more. Turn four adds three, turns five and six add three between them. If you are choosing a retry cap for a loop that returns real witnesses, this shape says four, and says the tail past that is not where your latency budget should go.
Here is the part that makes the paper honest about its own metric. In the full diagnostic run, every one of the 30 tasks passes its held-back test set by the final turn — a clean 100%. Then the authors go looking for the difference between passing and being correct: for each solved task they build a probe pool from short generic strings, the words in the task description, and the literal characters appearing in the reference and candidate patterns, and fire up to 320 probes per round at the pair. Any probe the two disagree on is a new counterexample.
Seven of the thirty survivors fail under probing, and the paper sorts the failures into three shapes. The one below is real: the pattern the model landed on for "a letter preceded by a number" passes the test set, and differs from the reference on strings the test set happened not to contain. Type a probe and see which side it separates.
The general lesson transfers straight to any agent loop with a test oracle: a fixed test set is both the feedback channel and the finish line, and an agent that gets to see the tests will write to them. The cheap defence is what the paper does — after the loop reports success, generate fresh inputs from the parts of the problem that are lying around, and re-check.
Assume a loop you already run: an agent produces something, a check runs, and on failure you send it back. The only change is what goes in the retry prompt. Apply the paper's measured per-turn curve to a volume of your own.
The arithmetic is illustrative — your tasks are not thirty pattern-synthesis problems, and your checker may not be able to produce a witness at all. But that last point is the one worth acting on: the swap only works if the failure is expressible. A test framework that reports "assertion failed" has thrown the witness away; one that reports the expected value, the actual value and the input that produced them has kept it. Which of those your agent sees is a logging decision, and it is worth more here than the model choice.