In short
- Reasoning models write out intermediate steps before they answer. Longer reasoning often helps on hard problems and wastes money on easy ones.
- Most systems use a fixed thinking budget. Adaptive systems learn when more thinking is likely to change the answer, and stop when it isn't.
- The same signal that decides when to stop can flag questions a model is likely to get wrong.
When a language model answers a question, it produces text one token at a time. Reasoning models are trained to produce a stretch of working first: intermediate steps, checks and false starts, often hidden from the user, before they commit to an answer. That working is sometimes called chain-of-thought, and the tokens it uses are often called thinking or reasoning tokens.
Working helps. On multi-step problems in maths, code and logic, models that reason before answering are much more accurate than models that answer straight away. But it isn't free. Every thinking token costs compute, adds latency and, with most providers, is billed like any other output.
Difficulty is uneven
Look at a set of questions one at a time and a pattern shows up. Many questions are easy: the model gets them right with little working, and extra thinking changes nothing. Some are too hard: the model gets them wrong however long it thinks. In between is a band where extra thinking turns a wrong answer into a right one.
A fixed thinking budget treats all three groups the same. It spends heavily on the easy questions, where it buys nothing, to reach the middle band, where it matters. On the hardest questions it buys a longer, more confident wrong answer.
Three ways to spend compute
- A fixed budget. Every question gets the same maximum amount of thinking. Simple and predictable, and wasteful on easy questions.
- Sampling and voting. The model answers the same question several times and the most common answer wins. It improves accuracy and multiplies the cost by the number of samples.
- Adaptive halting. The model decides as it goes whether to keep thinking. The idea goes back to work on adaptive computation time in neural networks (Graves, 2016). The hard part is training the decision to stop.
How a model learns when to stop
You can't directly label how long a question should take, because that depends on whether extra thinking would change the answer, which you only know afterwards. One practical approach is to measure it. For each training question, generate reasoning at several budgets, check whether the final answer is right at each, and look at the result: right at every budget, right only with more room, or wrong at every budget.
A small classifier, often called a halting head, can then learn to read the model's internal state at checkpoints during reasoning and predict whether continuing is likely to change the answer. When that prediction drops below a threshold, the model stops and answers.
A useful side effect
To decide whether more thinking will help, a halting head has to estimate whether the model is on track. A model that is lost can't be rescued by thinking longer, so the same signal can separate "I have this" from "I'm out of my depth" before an answer is written. That makes it useful for routing: low-confidence questions can go to a person or a larger model instead of getting a confident guess.
Where it breaks
- Rewording. A model's sense of difficulty is learned from the questions it was trained on. Phrase the same question differently and that sense can degrade faster than accuracy does.
- Easy-looking wrong answers. When a model finds a wrong answer quickly, it may stop early and sound sure. Telling "easy" from "easily wrong" is the hardest part of the problem.
- Latency. Adaptive budgets vary per question, so response times vary too. Services with hard time limits may prefer a fixed budget.
- Overthinking. Sometimes longer reasoning talks a model out of a correct first answer. More thinking isn't always safer.
What to ask of any reasoning model
- How are thinking tokens billed, and can you cap them?
- Can you see how long the model thought before each answer?
- Does it give a signal you can act on when it's unsure?
- How do accuracy and cost change as the budget changes, on your own questions?
Adaptive halting is the approach behind Kaer-R1, the model we're building now.