Every cheap-model-first cascade needs a rule for when to escalate, and almost all of them use a confidence number below a threshold somebody tuned by hand. That number carries no promise about how often you ship a wrong answer. This paper deletes the threshold and asks a different question: sample the small model 16 times, keep every answer that appeared often enough to clear a cut-off learned from a few hundred labelled examples, and accept the small model only if exactly one answer is left standing. The accepted answers then carry a provable error ceiling — and the rule beat the best hand-tuned heuristic on 49 of 72 model-and-benchmark pairings.
A cheap model answers a question sixteen times. The usual escalation rule takes the share of votes going to the winning answer and compares it against a threshold — 70%, say. But vote share collapses three genuinely different situations into one scalar. One answer dominating with a little noise around it is not the same shape as two answers splitting the vote evenly, and neither is the same as the votes smeared across every option. Step through them and watch a single threshold treat unlike things alike.
The threshold answers "did the winner clear a bar". The question you actually have is "how many answers can I not rule out" — and for a routing decision that second question is the useful one, because it is the one that can be tied to an error rate.
Here is the whole method. Score each candidate answer by how rarely it was sampled: an answer picked 12 of 16 times scores 0.25, one picked 16 times scores 0. Separately, take a couple of hundred questions whose answers you know, score the correct answer for each, and sort those scores. Pick the cut line at the position that leaves at most an α fraction of them above it — that is conformal prediction, a distribution-free way of turning a held-out sample into an error budget. Every answer scoring at or below the cut line goes into the set. If exactly one answer is in the set, the small model's answer is accepted; otherwise the query escalates.
Push α down to 0.05 and watch the cut line reach 1.00: because 8% of the calibration examples never sampled their correct answer at all, covering 95% of them forces the set to admit every option, and even the unanimous query escalates. That is the paper's real failure mode — when the small model's votes are too diffuse, the cascade defers everything and you have simply bought the expensive model with extra steps.
Standard split conformal prediction guarantees the set contains the correct answer at least 1−α of the time. Chaining tiers weakens that: each tier gets its own shot at accepting wrongly, so in the worst case a K-tier cascade's accepted-and-wrong rate is bounded by K·α — and if you want the cascade as a whole to hold at α, you calibrate each tier at α/K. There is a tighter reading. If coverage still holds conditional on which tier accepted — the paper calls this selection preservation — the bound collapses back to 1−α. That condition does not follow from split conformal, but the measured miscoverage tracks α rather than K·α, which is consistent with it holding in practice.
The honest reading: you are guaranteed K·α and you appear to get α. If a wrong accepted answer is expensive, calibrate at α/K and pay the extra deferrals — the guarantee is the thing you were buying.
Insisting on exactly one surviving answer is strict. Let two or three stand and far more queries settle at the cheap tier, so spend falls and accuracy falls with it — but now the trade is explicit and priced instead of buried in a threshold you nudged until the dashboard looked right. The paper's costs are per-call: the small model is 1 and the large one 2.7, so expected cost is 1 + 2.7 × P(escalate). Drag the acceptance rate and watch the frontier; the marked points are the paper's own measurements on a general-knowledge set.
Two anchors worth holding onto. At κ=1 the cascade edged the large model on accuracy while costing slightly more — you paid a little for a guarantee, not for accuracy. At κ=3 it cut expected cost 53% for 5.3 points. Loosening κ is the move; the accuracy is the receipt.
One detail the paper's cost units hide: the score comes from sampling the small model sixteen times. In their accounting a tier-1 call is 1, but in your bill it is sixteen small-model calls on every query, escalation or not. That overhead is the whole economics of this technique — it is free when the small model is an order of magnitude cheaper and it eats the saving when it isn't. Put your own prices in.
Two rules of thumb fall out. Sampling overhead is survivable only when the price gap between tiers is much larger than N, and the technique needs a finite answer set — so it lands on classification, routing, extraction and multiple-choice grading, not on free-form generation.