In short
- Calibration compares how confident a model is with how often it's right.
- Overconfident models are risky because their wrong answers sound as sure as their right ones.
- A well-calibrated confidence signal lets you send uncertain questions to a person.
Accuracy tells you how often a model is right. Calibration tells you whether you can tell when it's right. A model is well calibrated if, of all the answers it gives with 80% confidence, about 80% are correct; of those at 60%, about 60% are; and so on.
Why it matters
People check answers that sound uncertain and act on answers that sound sure. If a model is overconfident, its wrong answers arrive in the same assured voice as its right ones, so they don't get checked. Wherever people act on answers, in benefits, health or legal questions for example, a model's confidence is part of the product whether or not anyone measures it.
Calibration also makes a model more useful. If its confidence means something, you can set a threshold: answer automatically above it, and send everything below it to a person or a larger model.
How to measure it
- Reliability diagram. Group answers by stated confidence (50–60%, 60–70% and so on) and plot the actual accuracy of each group. A calibrated model sits close to the diagonal.
- Expected calibration error. The average gap between confidence and accuracy across those groups, weighted by how many answers fall in each. Lower is better.
- Brier score. The mean squared difference between confidence and outcome, where a right answer counts as 1 and a wrong one as 0. It rewards both accuracy and honest confidence.
- Error prediction (AUROC). How well a confidence score separates right answers from wrong ones. 0.5 is chance and 1.0 is perfect separation. This matters most if you plan to route on confidence.
Where confidence comes from
A language model's confidence can come from several places: the probability it assigns to its answer, a number it states when asked, how often repeated samples agree, or a separate signal such as a halting or verifier head. These don't always agree. A model can write "I'm fairly sure" while its internal probabilities say otherwise, so test the signal you actually plan to use.
Why models are often overconfident
Research on modern neural networks has repeatedly found them to be more confident than they are accurate. Training that rewards the answers people prefer can make this worse, because fluent, assured answers tend to be preferred. Training data matters too: text that is confidently wrong teaches a model to sound sure. Re-check calibration after any fine-tuning.
Practical steps
- Choose the confidence signal you'll use in production, and test that one.
- Measure it on your own questions, including reworded ones. Calibration measured on clean benchmarks often gets worse on real traffic.
- Set a threshold by weighing the cost of an unnecessary escalation against the cost of a confident wrong answer. Where the stakes are high, the second is usually far larger.
- Keep monitoring it. A drop in calibration is often an early sign that something has changed.