In an LLM self-test-and-repair loop, changing one line of the test-writer’s prompt — pasting the task specification as an enumerated rule checklist instead of saying “include edge cases” — raises final code correctness by +38 percentage points on every Claude tier, while cutting false alarms on correct code from 33% to 0%.
Models write code that nails the happy path and silently drops unstated edge cases: missing input checks, unhandled boundaries — e.g. Opus 4.8’s parse_range crashes on "1-2-3", "" and "-3-5" with opaque int() errors. The standard fix (CodeT, AlphaCodium, LEVER) is “generate tests, repair until green.” But why does that work? Two hypotheses with opposite advice: quantity — any plausible test sometimes trips a bug, so generate more — or grounding — tests derived from an external specification aim at the behaviours that matter.
The paper isolates the question to a single line. Both arms use the identical tester system prompt, the same tester model (Sonnet 4.6), the same budget (K tests, K = number of spec rules), the same repair loop, and a hand-written gold oracle no model ever sees. Flip the line yourself:
To write a test, the author must know the intended behaviour at each edge. An ungrounded tester must guess — twice. If it doesn’t think to probe an edge, the bug escapes (false negative). If it probes but guesses the wrong intended value, it rejects correct code (false positive). The enumerated spec supplies the missing information, so detection and precision improve together instead of trading off.
“spec” bundles three things: the spec’s content, its decomposition into rules, and a one-test-per-rule plan. Four conditions at fixed budget and tester pull them apart:
If the quantity hypothesis were right, doubling the test budget or unioning many independent suites should close the gap. Step through the arms:
In every arm, expected values come from the ticket or rules — never the candidate code — because a tester shown buggy code treats the bug as intended. The paper verified this directly by regenerating tests for 24 buggy one-shots with the candidate code pasted into the tester prompt:
Repair is only as good as the failing test’s expected value. On the six scale-up tasks detection ties (40/42 both arms) — yet final correctness is 96% vs 78%, because the ungrounded arm repairs toward its own wrong expectation. Step through the paper’s format_duration case:
On a HumanEval+ port (24 tasks, 5 model families) the gap vanishes: spec detects 13/22 vs free+ 14/22. Well-specified algorithmic benchmarks barely produce the bug class (pooled one-shot bug rate 6%), and 8 pure-logic tasks yielded 0 natural bugs in 45 valid submissions. The claim is bounded to specification-completeness defects — mostly small pure Python functions plus a stateful slice. The tester is deliberately strong (weak testers keep a 44–67% false-alarm floor even with rules), the repair loop is minimal, auto-derived specs recover only 47% of detection (edge-forcing recovers it but adds 33% false alarms — the binding cost is human knowledge of correct edge semantics), and the absolute 100% scores are artifacts of easy tasks: the stable quantity is the gap and its direction. No public code repository is released; the study is described as reproducible from task specs, reference implementations, gold suites and raw outputs.
If your product has a “write tests, then fix until green” step, this is a one-prompt-change experiment you can run this week. Walk the loop below and flip grounding on the test-writer step. Illustrative walkthrough — the code and tests are simulated; the per-task counts quoted are the paper’s.
Enumerate the spec as R1..RK checkable rules; one focused test per rule, in order; the tester never sees the code; every expected value computed from the rules. Rules about invalid input become error-expecting tests.
Practical notes from the paper’s sweeps: keep the spec version-controlled next to the ticket (dropping validation rules degrades detection visibly, never silently); don’t fully trust auto-derived rules (detection automates, precision doesn’t); and don’t score suites by mutation kills — mutation score actually favoured the ungrounded arm (95.9% vs 93.9%) because it rewards sensitivity regardless of whether the expected values are right.