Ask a model to write unit tests for a function and you naturally paste in the function. If it has a bug, you have just told the model what correct looks like — and it writes tests that agree. Across 11 models and 318 real Java defects, prompting with buggy code produced 8.4× more tests that pass on the broken version and fail on the fixed one, while suppressing the tests that would have caught the defect. Replace the code with a generated description of what the function is supposed to do and bug-finding tests rise from 104 to 187. Adding the description alongside the code does not work. It has to come out.
A test generator has to get its notion of correct behaviour from somewhere. Hand it the implementation and that is where it looks — so a defect in the implementation becomes the specification, and the tests come back asserting it. The damage runs both ways: more tests that lock the bug in, and fewer that would have found it. Switch what the model was shown.
Both harms at once. Showing the bug multiplies the tests that assert it by 8.4× and cuts the ones that catch it by 2.9×. This is not the model being sloppy — it is the model doing exactly what it was asked, against the wrong reference.
Earlier work treated any test passing on the buggy code as evidence of misguidance. That over-counts badly: most such tests simply exercise behaviour the bug does not touch. The paper insists on running every generated test against both versions, which splits the outcomes into four quadrants — and only one of them is the actual harm. Tap a quadrant.
Over 90% of the tests that pass on buggy code pass on the fixed version too — they are irrelevant to the defect, not misguided by it. If your evaluation counts them, it will report a huge problem and then fail to detect whether any fix helped, because the number is mostly noise.
The remedy is two prompts. First ask the model to write a docstring describing what the method is meant to do. Then generate the tests from that docstring with the implementation removed from the prompt entirely. The critical detail is the deletion: the paper tests all four combinations, and leaving the code in alongside the description performs slightly worse than code alone. Step through the modes.
The internal evidence closes the argument. Scored by three independent models, tests that assert the bug become more probable when the buggy code is in the prompt, and bug-catching tests become more probable when the fixed code is. The prompt is not just informing the model, it is moving what it prefers to write.
Most real pipelines do not generate once. They run the tests, feed failures back, and ask again. That loop amplifies whatever the first prompt was pointing at: with the buggy code in view, the extra rounds mostly manufacture more tests that agree with it. With a specification, the same rounds buy roughly twice as many real bug-catchers. Drag through the rounds.
Over three rounds the specification prompt adds 35.69 effective tests against 17.62 for the code prompt — and does it while accumulating 18.08 misguided tests against 31.77. Better on both axes, at exactly the point where a pipeline compounds its own mistakes.
The change is one extra model call per method before the one you already make. Set how many methods you generate tests for and how many carry a latent defect, and compare what lands in your suite. The per-method rates are the paper's; the volume is yours.
A test that asserts a bug is worse than no test: it turns a defect into a documented requirement, and the next person to fix it gets a red build for their trouble. That is the asymmetry that makes an extra call per method cheap.