Agent stacks are full of the same small component: embed two blocks of text, take the cosine, fire an alarm below a fixed cutoff. It is asked whether the meaning still holds. It answers how much of the wording changed — and those two questions point in opposite directions, because reversing an instruction is a one-word edit that leaves about 72% of the tokens in place while an honest restatement replaces nearly all of them. The production guard audited here caught 0 of 56 meaning-breaking edits and scored "withhold the study drug" against "administer the study drug" at 0.9608, on a cutoff that fires at 0.60.
The audited system is a drift guard: it re-embeds an instruction block on every run and alarms when similarity to the stored baseline drops. Below is the pair that made the paper, plus the two extremes of the corpus it was tested against — the most and least detectable meaning-breaking edit of the 56. Drag the cutoff. The five markers are real operating points, taken from the shipped system and from framework defaults the author opened and read.
The temptation is to read this as a badly tuned threshold. It is not. At the shipped 0.60 the guard also approved 15 of 15 faithful rewrites — the controls that should pass — so both classes sit in the same narrow band near the top of the scale. Raising the cutoff until the drug reversal at 0.9608 finally trips it takes you above the redundancy filter's 0.95 default, where honest rephrasings are being thrown away as duplicates. There is no setting where one class is above the line and the other is below it.
Here is the mechanism, and it is almost embarrassingly simple. The edits that change what an instruction tells you to do are small: insert a negation, downgrade a modal, move a decimal, drop a clause. The edits that preserve what it tells you to do are large, because saying the same thing differently means choosing different words. Cosine is sensitive to the second and nearly blind to the first. Step through the classes the paper tests and read both verdicts.
A score under 0.50 is not weakness, it is inversion: the gate is ranking the meaning-breaking edit as more similar than the faithful rewrite, more often than not. And the same embeddings are excellent at the thing they actually measure — detecting that the wording changed scores 0.975 to 1.000 across all nine configurations. The component works. It is answering a question nobody asked it.
If you build a test set the natural way — write some pairs that mean the same thing, write some that do not, measure — you get an answer that is about wording, because your two classes differ in wording as well as meaning. The paper builds its corpus factorially instead: every cell crosses the decision (same or opposite) with lexical overlap (close or distant), ten pairs per cell, so overlap is balanced inside each decision class. Flip the switch to see what that does.
So the encoders are not blind. Held at matched overlap they rank meaning well above chance — which is exactly why the result matters, because the gate you have shipped is still a coin flip. Step through the configurations. The two tasks are hypothesis distinctness (are these two research hypotheses the same claim) and constraint survival (does this rewritten instruction keep the constraint).
Two numbers that have to be held apart. Ranking ability is what the chart above shows, and the best encoder reaches 0.900. A shipped gate is a threshold on that ranking, and across all 90 configuration-threshold-task cells balanced accuracy ran from 0.450 to 0.700 with a median of 0.525. An encoder that ranks well can still have no cutoff worth deploying, because the two classes overlap where the cutoff has to sit.
This is the part that stops it being a tuning problem. The author tried the three repairs anyone would try, set a bar at 0.70 balanced accuracy in advance, and — critically — held out a second corpus written after the repairs were designed. Step through them, then flip to the held-out split.
The conditioned gate is the instructive failure. Adding word overlap as a second input is the right instinct — it is the confound, so subtracting it should help — and in-sample it clears the bar at 0.750. On text the author had not seen when the rule was written, it lands at 0.533. That gap is what an evaluation on your own examples buys you, and it is the reason the paper's headline recommendation is about who wrote your test set rather than which model you embed with.
The four use cases below are the real ones, with the thresholds they ship at. Pick one, set your volume, and set how often a near-identical pair actually flips the decision — the rate is yours to guess, but the catch rate is not: on this evidence, the gate finds essentially none of them. What comes out is the number of flipped decisions per day your gate signs off on.
The failure is quiet in a specific way. A gate that blocks too much generates tickets; a gate that approves everything generates a clean dashboard. Every one of these four had a green light on it for the whole audit.