In short
- Sampling settings such as temperature add deliberate randomness.
- Small changes to wording, order or context can change an answer as much as switching models.
- Serving infrastructure and model updates add variation you don't control, so pin versions and test across phrasings.
A calculator gives the same answer every time. A language model often doesn't. Sometimes that's intended, and sometimes it's a side effect you need to manage. There are four main causes.
1. Sampling
At each step a model assigns a probability to every possible next token, and settings such as temperature and top-p decide how it picks between them. At temperature 0 it takes the most likely token each time. At higher temperatures it samples more widely, so the same prompt can produce different answers. Higher temperatures suit creative work. For factual or procedural answers, a low temperature is usually the right default.
2. Wording
Models are sensitive to how a question is phrased. Putting the background after the question instead of before it, merging two questions into one sentence, or using a slightly wrong technical term can all change the answer, sometimes by as much as switching to a different model. Typos tend to matter less than people expect; order and structure tend to matter more.
This matters because you don't control how your users write. A benchmark score measures one phrasing, and real traffic contains many.
3. Context
Everything the model sees is part of the question: the system prompt, earlier turns in the conversation, retrieved documents and tool results. Change any of them, by editing the system prompt or retrieving a different document, and the answer can change even when the user's words are identical.
4. Serving and versions
Even at temperature 0, outputs aren't always identical. Hosted models run on shared hardware that processes requests in batches, and tiny numerical differences can change which token wins when two are nearly tied. Providers also update their models, and an alias such as "latest" will point to a new version without your code changing at all.
What to do about it
- Set sampling deliberately. Use low temperatures for answers people act on, and record the settings with every call.
- Pin the model version. Use a dated snapshot rather than an alias, and plan for its retirement.
- Test across phrasings. Write several versions of each test question and look at the spread, not just the average.
- Structure your inputs. If your product collects a user's situation and their question, ask for the situation first and the question last.
- Log what was served. Keep the exact prompt, settings, model version and response for anything that matters, so you can explain an answer later.
When variation is useful
Disagreement between repeated answers can be a signal. If a model gives different answers to the same question, that question is a good candidate for review by a person. Some systems sample several answers on purpose and treat disagreement as a sign of uncertainty.