Take a question a model gets right. Change nothing but the wording. How often does the answer break? We ran that at proper scale: 40 base questions, each rewritten into 12 paraphrase families, put to 312 models at temperature zero, with five seeds on the sampled variants. That is 480 phrasings per model and 149,760 scored runs.
The median model's accuracy moves 19 percentage points between its strongest and weakest paraphrase family. In our reliability ranking, the gap between a model and the one two places above it is about 11 points. For the median buyer, wording matters more than which of three adjacent models they licensed.
The half of the problem you don't control
Benchmark scores are known to be brittle, and 2024 was the year the field documented it. A fresh set of grade-school maths problems, written to match a saturated one, pulled down several models that had aced the original, the sign of a suite partly memorised rather than reasoned through. The same model run through two open evaluation harnesses can post different numbers on one benchmark, most of the gap down to the template and answer parsing. MMLU, the field's most-cited knowledge test, is widely known to carry mislabelled and ambiguous items. Human-preference leaderboards added style controls once it was clear voters rewarded longer, more formatted answers over better ones.
Every one of those failures is the evaluator's to fix. The template that frames the question and the code that grades the answer are a lab's to choose. The UK's AI Safety Institute open-sourcing its Inspect framework is part of that repair, giving evaluators a shared, inspectable harness. Our study measures the part nobody owns.
Format variance is the lab's problem. Phrasing variance is the user's, and no amount of harness hygiene touches it.
The twelve families
Each base question was rewritten by hand into twelve styles we see constantly in live traffic. Why by hand, I will come to.
- Formal register, full sentences (the benchmark original)
- Compressed (
visa hours france student?) - Typo-laden, thumbed out on a phone
- Common ESL constructions, dropped articles and flattened tense
- Context after the question (
Can I appeal? Background: ...) - Context before the question, the ask buried at the end
- Two questions tangled into one sentence
- Emotionally loaded (
I'm panicking a bit) - Over-polite and heavily hedged
- Terse imperative (
give me the deadline rule) - Irrelevant detail folded in, an aunt's opinion or the weather
- Wrong technical vocabulary, used with confidence
What moved, and what didn't
Typos barely register. Across the fleet the typo family costs 2.4 points against the formal original. We had budgeted it as a main effect and were wrong; models have plainly read enough messy text to look straight through it.
Word order is where the damage sits. Context after the question is the worst family for 61% of models, at a median cost of 14 points. A model commits early and revises reluctantly once the facts arrive too late. The wrong-vocabulary family does something odder: it splits the fleet almost in half. About a third silently correct the user's mistaken term and answer what they meant; the rest answer the question as literally worded, usually not the one that mattered. Those are two different products, and no spec sheet tells you which you have bought.
| Paraphrase family | Median Δ vs formal | Worst family for % of models |
|---|---|---|
| Context after question | −14.1 pts | 61% |
| Two questions tangled | −9.8 pts | 17% |
| Wrong vocabulary | −7.2 pts | 12% |
| Compressed | −4.6 pts | 6% |
| Emotionally loaded | −3.1 pts | 3% |
| Typos | −2.4 pts | 1% |
Two null results are worth recording. Politeness and hedging cost nothing measurable, inside a ±1 point band indistinguishable from noise. The emotional-load penalty is real at 3 points but smaller than folk wisdom claims. When a panicking user gets a worse answer, most of the damage is the tangled questions and buried context that panic produces, not the feeling.
Method, with the awkward parts left in
Four people wrote the paraphrases and a fifth checked each for answer preservation; any variant where the correct answer arguably shifted was cut, 11 of 491 drafts. We did not use a model to generate paraphrases, because paraphrase style is the exact thing under test. Machine-written variants cluster in model-idiom space and flatter the models being evaluated, so you would be measuring how well a model handles its own dialect. Doing it by hand cost about two weeks.
Scoring was exact-match where the format allowed, otherwise a three-judge panel of pinned open-weight models with a human breaking ties. Human adjudication was needed on 7.4% of graded items, and inter-judge agreement ran at α = 0.83. Judges never saw which family produced a response, so none could mark an answer down for arriving in broken English.
Two build decisions and a buying rule
The first decision is nearly free. If your product collects a user's situation and their question in one form, put the question field after the context field. That ordering avoids the context-after-question penalty, worth more at a median of 14 points than most model upgrades this year.
The buying rule follows from the spread. A single benchmark score is one draw from a distribution 19 points wide; treat it the way you would treat a survey with one respondent. When you assess a vendor, ask for the score across a range of phrasings rather than one clean prompt, and if they cannot supply it, run your own worst-phrasing users past the model before signing.
The rest is why single-prompt benchmarks overfit. Score every model on one fixed phrasing and you reward the ones that sit well with it, and across releases the field drifts toward the house style of its favourite tests. That is the mechanism the grade-school maths replication exposed, except there the fixed thing was the question set and here it is the wording.
What we can't claim
Our 40 base questions lean civic and administrative, because that is where our partners deploy. A maths-heavy or code-heavy battery might order the families differently, since word order may matter less where the payload is a formula. We ran each model once, so the family rankings are stable while the per-model figures carry the usual single-run caution. The full phrasing set and protocol go to anyone running the extension: no licence fee, one email.
Every score in our Model Atlas now prints its paraphrase spread beside it. The change makes previously impressive models look ordinary, and it makes two dull-looking ones look genuinely steady across the full range of phrasings.