When you compare two coding models by pass rate, you are also comparing whatever wrapper ran them — the layer that issues tools, manages context, and decides when to stop. Hold the model fixed and swap only that wrapper, and pass rate moves 0-8 points, mostly inside the noise. Tokens burned per solved task moves up to 40×: about 28,000 for the leanest pairing against roughly 1.5 million for the heaviest. Each wrapper also fails in its own signature way, and the signature repeats across both models — so it belongs to the scaffold, not the model you were trying to judge.
Six harness-model pairs, one benchmark, one task set. Rank them by pass rate and they cluster inside twelve points of each other — you would call it a wash and pick on taste. Rank the same six by what they burned to get there and the table falls apart. Switch the sort and watch the order change.
Pass rate spans 38% to 50% — a spread the paper's paired confidence intervals mostly cannot separate from zero. Cost spans two orders of magnitude. The leaderboard reports the first column and stays quiet about the second.
The clean way to isolate the scaffold is to fix the model and vary only the wrapper. Do that and there is nowhere left for the difference to hide: it is the harness deciding how much context to pre-load, how many times to re-read it, and how long to keep going. Pick a model and compare its three wrappers.
One wrapper pre-loads the file tree eagerly and stops early; another keeps a persistent tool loop running until the clock expires. Same model, same tasks, same prompt intent — forty times the tokens for a result you cannot statistically distinguish.
The failures are not randomly distributed. One harness stops voluntarily and says it cannot do the task. One declares itself finished on work that fails its own verification. One spins until the wall clock kills it. Crucially the pattern holds when you swap the model underneath — which is what makes it a property of the scaffold rather than the model. Step through the wrappers.
Flip the model switch and the shape barely changes. That is the whole argument: a failure mode that survives a model swap is telling you about the wrapper, and it will follow you to whatever model you migrate to next.
The paper counts turns where the agent neither edited a file nor ran a command — a full round-trip to the model that produced no action. These cost tokens, but the sharper cost is time: if a person is sitting in front of the agent, every idle turn is dead air. One wrapper averages ten times more of them than another, and again it replicates across both models.
A no-action turn is not just a token tax. It is latency the user watches, and oversight burden for whoever is supervising the run — the paper's argument for reporting wall-clock and turn counts next to any pass rate you publish.
The paper's recommendation is not "use this harness". It is that the unit of selection is the harness-model pair, chosen under a cost or latency budget rather than by pass rate alone. Set a per-task token ceiling and see which pairs survive it — and what you give up in pass rate by insisting on the cheap ones.
If you publish agent comparisons, publish the harness spec, the token totals and the wall-clock beside the pass rate. If you consume them, assume any number quoted without a harness is describing a system you have not seen.