Everything else about an adaptive-compute deployment is configuration. The halting threshold is the design decision: it sets what you spend, how often the model stops before it should, and how much traffic reaches your fallback path. This is how to set it from your own data rather than from a default.
What you are trading
Raise the threshold and the model thinks longer: more cost, more latency, fewer under-thought errors. Lower it and the reverse. The curve is not symmetric, and its shape is a property of your traffic — specifically, of what fraction of your questions sit in the band where extra deliberation actually changes the answer. In our hard-reasoning suite that band is about 19% of items. In a support-triage workload we instrumented it was under 6%, and the correct threshold was far lower as a result.
The method
- Sample 400 real requests, stratified across your domains, not cherry-picked. Consented and anonymised.
- Run each at five fixed budgets — 64, 192, 512, 1,536 and 4,096 thinking tokens — and record correctness at each. You now have a budget-response curve per item.
- Classify the curves. Three shapes cover most traffic:
Curve Meaning Right action Flat-high Right at every budget Stop immediately — thinking is pure cost Rising Wrong when rushed, right with room Spend. This band is the entire prize Flat-low Wrong at every budget Stop early and escalate - Sweep the threshold across the sample and plot cost per solved task against your harmed-rate.
- Pick the knee, weighted by what a wrong answer actually costs you.
Steps 1–4 are an afternoon. The whole exercise costs roughly what the £40 behavioural read costs, because it is essentially the same machinery.
Weighting the asymmetry
Most teams set the threshold to minimise cost subject to holding accuracy, which is the wrong objective. The right one accounts for the fact that a wasted escalation costs you a few minutes of someone's time, while a confident wrong answer on a deadline question can cost the user their claim.
A workable rule: estimate the cost ratio between those two outcomes for your domain, and set the threshold so the expected cost is minimised rather than the direct spend. In welfare and clinical contexts we have seen ratios above 100:1, which pushes the threshold much higher than a naive cost sweep suggests.
The non-monotone cases
About 6% of our training corpus is non-monotone: right at 192 tokens, wrong at 1,536, because a longer chain talks the model out of a correct first instinct. Do not average these away. If they are common in your traffic — they cluster in questions with a strong correct prior and a plausible distractor — a higher threshold will make things worse, not better, and only the sweep will tell you.
Watch it after you set it
Log the halting-score distribution and alert on shape changes, not just on the mean. A second hump appearing is the clearest signal that something upstream moved — a model build, a prompt edit, a retrieval change. It is the same lesson as our harness post-mortem: a bimodal histogram where there should be one mode means two things are being counted as one.