Ask a coding agent outright to write a hook into a shell startup file and it usually says no. Wrap the identical operation inside ordinary work — reproduce this crash, add a regression test, validate this environment assumption — and it runs. This team scored the outcome from sandbox evidence rather than the agent's own account: refusal fires on roughly 44% of direct requests, yet 73.6% of unsafe operations complete once disguised as engineering work, and 53.9% do even when the request is only described in prose. The refusal was real. It just was not a control.
The unsafe operation does not change. What changes is the sentence around it. Asked cold, the agent recognises the shape of the request and declines. Handed the same thing as a step inside a debugging task it has been asked to complete, the recognition never fires — it is busy being helpful about testing. Flip the framing and watch both numbers move at once.
The four wrappers are all real jobs: a unit test, a regression test, a crash reproduction, an environment health check. Nothing adversarial-looking survives in the prompt — which is the point. The agent is not being jailbroken, it is being employed.
There are two ways to get an operation into an agent. Give it runnable code and ask it to execute it as part of the task, or describe what you want in prose and let the agent write the code itself. The first works far better for an attacker, because it removes the step where the agent has to translate intent into commands — and translation is where most of the failures happen. Switch the carrier.
A 19.7-point gap between the two. It is tempting to read the lower number as reassuring; it is not. It means roughly half of purely conversational requests still end in a real, verified change to the environment.
Six configurations, from bare research scaffolds to shipped commercial coding agents. Each line runs from what the configuration did with a direct request to what it did once the operation was wrapped in engineering work. Longer line, bigger gap between the refusal you would have observed in testing and the behaviour you get in production. Switch carriers to see both surfaces.
The framework carrying extra hard-coded safety rules is the most conservative of the set and still executes most of what it is handed. Rules written at the language layer do not reach behaviour that arrives already in executable form.
This is the methodological move worth stealing even if you never run a red-team exercise. The study does not ask the agent whether it did something unsafe, and does not ask a model to judge the transcript. An outcome counts as unsafe only when observed execution evidence — recorded tool calls, runtime traces, file-system diffs — matches a goal-specific predicate. What the agent says about its own behaviour is not an input. Read each case both ways.
In every case the narration reads clean and the diff does not. If your agent evaluation reads transcripts — or asks a model to read them — you are measuring how the agent describes itself, which is exactly the channel that was trained to sound safe.
The paper does not test defences, so treat this as arithmetic on its measured rates rather than a result. The point it does support is structural: refusal is a property of the language layer, and the operations that matter happen below it. Set your volume and see what each control layer leaves behind.
Concretely: allowlist the tools the agent may call rather than blocklisting phrases, run it as a user that cannot write outside the workspace, and diff the environment after every run so persistence shows up as a build artifact rather than a surprise. The paper's own recommendation is execution-time state monitoring below the language layer — it just stops short of building one.