Your system prompt is fighting itself

Four papers today about the parts of an LLM system nobody tunes: the list of rules you stack in the system message, the moment you decide a run is not going to work, the argument fields of a tool call, and the shape of the generation request itself. In each case the failure is mechanical rather than a gap in what the model knows, and in each case the remedy is something far smaller than a better model — a one-off rewrite of the prompt, a 0.6-billion-parameter classifier watching the transcript, a logistic regression on one layer of activations, or simply refusing to ask for the whole document in a single call. None of the four would be fixed by upgrading the model.

Today's pick

96% → 20%

how often a model obeys a given rule as one system message goes from one rule to twenty

Instruction Stacking Collapse: A Benchmark and the Capability-Dependent Value of Prompt Compilation

A production system message asking for valid JSON, a word cap, three citations and a fixed tone is four instructions, and models obey each one far less often together than alone. Stacking twenty machine-checked rules drops the per-rule follow rate from about 96% to 60.4% on Claude Sonnet 4.6, 43.3% on Gemini 2.5 Flash and 20.0% on GPT-5-mini — and the collapse is structured, not noise: 15 pairs of these rules cannot both be satisfied, and "output valid JSON" alone is incompatible with nine others, so one format demand silently voids the headings and the "Summary:" line you also asked for. The training-free fix is an instruction compiler — a single model call that regroups the stack into a precedence-annotated checklist, computed once per prompt and reused across queries — and its value is capability-graded: 11 points of follow rate back on the weakest model, nothing measurable on the strongest. Audit your longest system prompt for pairs that cannot both hold before you spend another day tuning wording.

open the explainer → abstract PDF

+5.2 pts at 14–20% fewer tokens

more repository issues resolved, from aborting a run that is going to fail and starting it over with the abandoned edits on offer

Fail-Fast, Restart-Smart: Early Failure Prediction and Restart for SWE Agentic Tasks

A coding agent that is going to fail usually fails slowly: the trajectory stretches, exploration starts looping, and tokens burn until it gives up. This paper trains a 0.6-billion-parameter monitor that reads only what is visible — the issue text plus the last eight steps of thought, action and observation, no logits and no hidden states — and calls the run doomed mid-flight; trained on one policy's trajectories it transfers unchanged to three others including a hosted model, saving 14.6% to 20.4% of execution tokens at a 5% false-alarm rate. When it fires, the agent restarts with no prompt history but with the abandoned repository diff offered as an overlay it can inspect, apply or discard, lifting resolution from 66.6% to 71.8% where restarting from nothing reached only 66.8%. The transferable move is separating the two things a retry throws away: dump the poisoned context, keep the code that was already written.

open the explainer → abstract PDF

19.7% → 59.6%

of tool calls with every argument exactly right, averaged over five open models and seven test sets

Getting the Parameters Right: A Difficulty-Graded Benchmark and Probe-Guided Training for LLM Tool Calls

Tool-use research has gone almost entirely into choosing the right tool and ordering the calls; filling in the arguments was left to hope, and on real cloud-network APIs even frontier models get fewer than half of them exactly right — one scores 29.4% with no examples. The finding worth borrowing is that the model already knows it is about to be wrong: a logistic regression on the hidden state captured just before it writes each parameter value predicts that value's correctness at 0.986 AUC, well clear of the 0.914 you get from the token probabilities. Using that score to filter self-generated training data and to rerank sampled candidates moves average exact match from 19.7% to 59.6% — though the decomposition is worth stating plainly, since ordinary fine-tuning already reaches 51.6 and the probe adds the last 8.0. It needs hidden-state access, and a probe refitted per model, dataset and sampling temperature, so this is for teams serving their own weights.

open the explainer → abstract PDF

+15 to +63 pts

more workflows that actually validate, from splitting one generation call into ordered stages — every model improved

Evaluating LLM Trade-offs for Enterprise Automation: Lessons from Workflow Generation in a Production Enterprise Platform

This is a deployment report rather than a lab study: six models generating real automation workflows on a live enterprise platform, across 29 scenarios taken from actual user requests, eight runs each, 2,784 runs in total. The first architecture asked a single model call to declare typed variables, instantiate top-level blocks, nest control structures and wire inputs to outputs all at once, and only 31.5% to 82.8% of its output was structurally valid. Splitting that into ordered stages — plan first, gather evidence, reconcile the plan, then generate variables, then top-level blocks, then nested blocks — lifted every one of the six models, to between 74.1% and 97.8%, and made a 24B model at one cent per workflow a serious production option against one costing 19 times more for 0.4 points of extra validity. The warning sits in the same tables: the model with the best structural score satisfied only 6.9% of prompts on human review, because it had learned to emit a schema-valid stub announcing the integration was unavailable.

open the explainer → abstract PDF