A skill is a written procedure you hand an agent — a runbook, an instruction file, a set of steps in the system prompt. It is described to the runtime and never encoded in it, so the model re-derives the control flow on every run and quietly drops the mechanical parts. Across 30 such procedures and 1,080 runs, the prose agent performed 56% of the steps its own file mandated while still shipping artifacts that passed the output checks. This paper compiles the prose into a program: every mandatory step becomes a gate that fires when execution enters the node, every genuine judgement stays a typed slot the model fills. Compliance goes to 86%, whole procedures finish 2.3× as often, and it costs 0.58× the tokens. The part that should worry you: on a newer model the prose agent climbed to 68% and the harness did not move at all.
These failures are hard to notice because the deliverable looks right. The agent produces a document, a status report, a resolved issue — and a reviewer checking the output signs it off. What is missing is in the middle: the command that was supposed to run, the request that was supposed to be made, the pause that was supposed to happen. Step through three of the paper's own skills and switch the arm.
Note what these three have in common. Not one is a comprehension failure — the model can quote the rule back to you. The paper's own reading: the deficiency is in execution, not comprehension, and the misses concentrate on the mechanical steps rather than the judgement calls.
The denominator matters here, and it is honest: only mandates the run actually reached are counted. A step skipped because the agent correctly branched away from it is scored not-applicable, not missed. So the 44 points the prose agent leaves on the table are steps it arrived at and did not take.
The whole method rests on a single test applied to every line of the procedure: is this step's output a function of its inputs? If it is — the test suite decides whether tests pass, the API decides what the issue says, the filesystem decides whether the commit landed — then code owns the step and it should never have been a request to a model. If it is not, because the step needs taste or open synthesis or a call the inputs do not determine, the model owns it and stays in charge. Try it on some real steps.
The force of the rule matters as much as its owner. A must becomes an ability that fires the moment execution enters the node — it cannot be reasoned past. A must not becomes nothing at all: the path is simply absent from the compiled program, which is the strongest guarantee available and the cheapest. A may stays a typed verdict the model returns.
The obvious way to build this is to ask a model to turn the prose into a program in one shot, which fails in the way you would expect: it invents obligations that were never in the file and forgets ones that were. The pipeline instead alternates — the model proposes, code disposes — and then freezes the rule set before anything is generated. After the freeze nothing downstream can add a rule, drop one, or soften it. Step through the stages and watch where the model is allowed to speak.
Two properties fall out of that ordering. Every element of the compiled program traces back to the sentence that required it, so you can audit the harness against the file it came from. And the last stage runs with no model call at all — lowering is deterministic, which is why the same compiled artifact runs unchanged on either model.
This is the number that surprised me most. Compiling does not mean handing the procedure to code — across all 30 skills only 37% of mandated steps become gates, and the remaining 63% stay model-owned slots. The lift comes from moving a minority of steps: specifically the mechanical ones the prose agent was reporting as done without doing.
The natural objection is that this is a temporary problem — today's model skips steps, next year's will not, so why build a compiler. The paper ran both arms on two model generations and the answer is uncomfortable. The prose agent does improve, substantially. The harness does not improve, because it was already doing the thing. Switch the generation and watch which bar moves.
Read the two bars as two different kinds of claim. The prose number is a behaviour — it is whatever the current model happens to do, and it will move again next release, in a direction nobody controls. The harness number is a property of the program. That is the actual argument for compiling: not that it scores higher today, but that it is the only one of the two you can make a promise about.
Take your own longest instruction file and split it in two: the steps whose outcome is decided by something outside the model — run this, fetch that, wait for approval, write the file — and the steps that need judgement. Set the shape below. The per-step rates are assumptions, tuned so that at the paper's own 37% mechanical share they reproduce its measured 56% and 86%; the split between them is where the insight is, not the absolute numbers.
Push the mechanical share up and the prose bar collapses while the compiled one barely notices — which tells you which procedures are worth compiling first. A skill that is mostly taste loses little to prose. A release checklist, a compliance routine, a verify-then-report step: those are almost entirely steps whose outcome is decided elsewhere, and those are exactly the ones being skipped in a way the deliverable does not reveal.
One honesty note on this widget: it deliberately does not predict whole-procedure completion. Per-step rates do not multiply out to the paper's measured 28% and 65%, because misses cluster inside a run rather than landing independently — a run that skips one mechanical step tends to skip the next.