In short
- Compare models on cost per solved task: total spend divided by questions answered correctly.
- Thinking tokens, retries, long prompts and escalations to people often cost more than the headline price suggests.
- A cheaper model with a reliable "I'm not sure" signal can beat a more accurate one on total cost.
Model prices are usually quoted per million tokens, with separate rates for input and output. That's useful for comparing providers, and it tells you very little about what a deployment will cost.
Where the money goes
- Input tokens. System prompts, conversation history and retrieved documents are sent with every request, so a long system prompt is paid for on every call.
- Output tokens. Usually priced higher than input. Longer answers cost more.
- Thinking tokens. Reasoning models can produce a lot of hidden working before they answer, and it is typically billed as output even though you don't see it.
- Retries and sampling. Asking several times and taking a vote multiplies the cost.
- People. Every answer that a member of staff escalates, checks or corrects has a cost, often far larger than the model call.
Cost per solved task
One number cuts through this. Take the total cost of handling a set of real tasks, including model calls and staff time, and divide it by the number handled correctly. Measure it on your own traffic, not on a benchmark.
A worked example with round numbers. Model A costs £2 per 1,000 questions and answers 70% of them correctly. Model B costs £6 per 1,000 and answers 85% correctly. On model costs alone, A is cheaper per correct answer. Now suppose each wrong answer takes £1 of staff time to catch and fix. Across 1,000 questions, A leaves 300 wrong answers and B leaves 150, so A costs £302 in total and B costs £156. The cheaper model only wins if its mistakes are cheap to catch.
Why "I'm not sure" saves money
If a model can reliably flag the questions it's likely to get wrong, you only need to send those to a person, and everything else can be answered automatically. A cheaper model with a good confidence signal can then beat a more accurate model that sounds equally sure about everything. When comparing models, test the confidence signal as well as the accuracy. Our explainer on calibration covers how.
Ways to reduce cost
- Use batch processing for work that doesn't need an instant answer. Many providers discount batch requests, often by around half.
- Cache repeated prompts and responses where your provider supports it.
- Keep system prompts short and send only the context each request needs.
- Cap thinking budgets, or use models that stop early on easy questions.
- Send simple questions to a smaller model and escalate the rest.
Report cost with every result
An accuracy figure without a cost next to it is incomplete. When you compare models or report on a pilot, put cost per solved task next to accuracy, and include the cost of the people who handle what the model can't.