A coding agent writes one line of shell. Between the model and the shell sits an interface that may serialise, wrap and re-read it — a remote wrapper, a container exec, a CI run step. Replaying the identical stored reply through one added double-quoted parser cost every one of eight model configurations 55.4 to 73.2 points of task success, on 56 tasks validated by exact final state. Telling the model where its reply would run recovered 30.4 to 60.7 points for six of them — so a single matched score can sit near zero while hiding both effects at once.
The benchmark's tasks are ordinary agent work: write exact bytes to a file, pass a literal argument, edit a JSON file, handle a file whose name starts with a dash. Every one is graded on the final state — the bytes on disk, the argument vector received, the parsed JSON — so any command that reaches the target state gets credit. Step through the families and flip the transport. Nothing about the model's reply changes when you flip it; only what happens to it in the pipe does.
The failure is not confined to adversarial payloads. The 14 benign control tasks — one per family, no hostile characters at all — lose 28.6 to 57.1 points on their own, because models emit dollar signs, backticks and quotes in perfectly ordinary commands. And you cannot see it from the return code: across the failing executions, 23.4% to 47.0% exited zero while leaving the wrong state, so a harness that trusts exit status misses up to half of these.
Here is the design that makes this measurable. Generation contract — what the model is told about how its reply will be used — is crossed with execution transport — what actually happens to the reply. Four cells. Replay uses the stored reply, so no new model calls are involved and the added parser is the only thing that changed. Step through the configurations.
The strongest configuration is the clearest case. Its matched score — written for the boundary, run through the boundary — is 3.6 points below its direct-path score, which reads like a rounding error. Underneath it, the added parser destroys 64.3 points and the boundary-aware rewrite restores 60.7. Two configurations do not adapt at all, and for them the matched gap is the damage, undisguised.
The boundary-aware prompt differs from the plain one by a single sentence stating where the reply will be interpolated. It gives no quoting advice. That alone is what capable models act on — and the adaptation is specific to the grammar they were told about, not a general hardening. Switch which wrapper the model is told about, then read the two columns: it passes on the wrapper it was promised and fails on the other.
Any pipeline that transforms generated output before executing it defines the same four cells. The paper tests a second one: embedding the same replies into JSON by hand, the way a hastily written tool-call bridge does. The serialiser re-reads exactly the characters the shell does. Pick a transform and see what the identical set of replies scores.
Both repairs are trivial, and that is the paper's own point: the contribution is the measurement, not the fix. Escaping the reply at the interpolation point reproduced the direct-path outcome for all 448 replay pairs; handing it over as a temporary script did the same for all 448 public and 126 private ones. Neither repairs a genuinely wrong command — the 33 that fail on the direct path stay failed.
Pick the shape of your execution path, set how many commands your agents issue in a day, and read off how many arrive broken. The retention figure is the measured one — reparsing preserved 123 of 415 direct-path successes across the campaign, a per-configuration retention of 25% to 35% — and the volume is yours.
And the ranking consequence, which is the part that touches procurement: selecting a model by direct-path success picks one that passes all 56 tasks directly and 50 of 56 on the wrapped path, while the path-aware pick reaches 51. The regret is one task at a saturated frontier — but the top rank reverses, and further down the ladder the reordering is much larger.