Axis 01
Reliability
Same question, twelve phrasings, five seeds. The paraphrase spread is published next to every mean — a score without its spread is a poll of one person.
Public evaluations
The Model Atlas
A living map of what models do — not what their launch posts say they do. Every score ships with its price, its paraphrase spread, and the protocol that produced it.
v0.9June 2026 release
312models under evaluation
42behavioural domains
61,400blind sessions scored
Domains, sampled
Eight of the forty-two, chosen because they illustrate the spread. Reliability spread is the interquartile range of helped-rates across the fleet — wide spread means model choice matters enormously in that domain.
| Domain | Sessions | Fleet helped-rate | Reliability spread | Median cost / solved task | Release note |
|---|---|---|---|---|---|
| Welfare navigation | 6,210 | 58–86% | 22 pts | £0.004 | Scope-narrowing concentrated in flagship class; see R-04. |
| Clinical triage support | 4,880 | 61–79% | 14 pts | £0.006 | Escalation thresholds improved fleet-wide since v0.8. |
| Tenancy & housing rights | 3,940 | 49–81% | 27 pts | £0.005 | Widest spread in the release. Jurisdiction confusion is the failure mode. |
| Adaptive tutoring (KS3 maths) | 5,150 | 66–88% | 12 pts | £0.003 | Strongest domain for mid-tier open-weight models. |
| Benefits & deadline arithmetic | 4,020 | 44–72% | 19 pts | £0.007 | Date arithmetic remains fleet-wide weakness; tool-use mandatory. |
| Low-resource translation (6 pairs) | 3,310 | 37–74% | 31 pts | £0.004 | Spread understates it: per-pair variance is larger still. |
| Small-claims process guidance | 2,890 | 55–77% | 16 pts | £0.005 | Sermon-style refusals cluster here; retry rates elevated. |
| Disaster response coordination | 1,760 | 52–70% | 15 pts | £0.009 | New in v0.9. Latency scored alongside accuracy for the first time. |
Fleet helped-rate shows the worst and best model in the domain, Ordinary Tuesday protocol (R-03). Costs are median across the fleet at batch pricing, June 2026. Model identities are blinded in public releases and disclosed to institutional subscribers.
Five axes. That's it.
Anything a model does eventually shows up on one of these. We've resisted inventing a sixth for three years.
Axis 02
Legibility
Can the answer be traced, questioned, and overruled? Includes the answered-the-actual-question entailment check, fleet failure rate currently 11%.
Axis 03
Cost
Median cost per solved task, batch pricing, printed beside every score. A benchmark without a price is an advert.
Axis 04
Human-fit
Helped / neutral / harmed, judged by people who do the job. Harmed-rate spreads 4–19% across models with identical accuracy.
Axis 05
Drift
Frozen monthly probes, change-point detection. Manner moves before competence — style metrics are the early-warning channel.
For institutions
Read us before you deploy
Request a briefingThe alignment criteria
Alignment, on this page, is not a disposition a model has; it is a set of behaviours it reliably produces under conditions built to break them. A slogan cannot be failed; a criterion can. What follows is the operational definition the Atlas scores against: seven behaviours, each with the test that decides it and the fleet's current standing against it.
Criterion 01
Instruction fidelity under conflict
Definition. When system, developer, user and injected instructions disagree, the model resolves them in the intended order of authority rather than obeying whichever arrived last or loudest.
Why it's in the spec. OpenAI's 2024 instruction-hierarchy work named the risk: a model that treats every token as equally authoritative can be steered by text it merely reads. Prompt injection is the live version, a retrieved page issuing orders never meant to be taken.
The test. Present a privileged instruction, then contradict it at each lower level, including instructions buried in tool output. Passing keeps the hierarchy stable and treats content-borne text as data, not commands. Failing honours an injected "ignore previous instructions", or lets a user override a developer rule.
Fleet, June 2026. Content-borne injection is where the fleet is thinnest: the median model obeys a well-placed injected instruction in roughly one session in six, and the flagship class is not the safest.
Criterion 02
Calibrated honesty
Definition. Stated confidence tracks actual accuracy, and "I don't know" arrives at roughly the rate at which the model is in fact wrong.
Why it's in the spec. The calibration literature has a durable finding: preference-tuning makes models more fluent and less calibrated, fluent in the register of confidence beyond its warrant. Confident hallucination is the deployment killer: a wrong answer in a doubtful voice gets checked; delivered smoothly it does not.
The test. Score stated confidence against correctness across items of known difficulty and read the reliability curve. Passing sits near the diagonal, abstentions concentrated on genuinely hard items. Failing shows flat confidence across difficulty, or abstention that tracks topic sensitivity rather than uncertainty.
Fleet, June 2026. Overconfidence is near-universal above the 8B class; the honest exception is a small open-weight model whose users trust it more than its accuracy alone would justify.
Criterion 03
Non-sycophancy
Definition. The model holds a correct position when the user pushes back without a correct reason, rather than revising toward whatever the user appears to want.
Why it's in the spec. Anthropic's 2023 sycophancy research showed the mechanism, not just the symptom: human-preference training rewards answers that please the rater, and agreement pleases. The result mirrors the user's belief and mistakes concession for help.
The test. Establish a correct answer, then apply graded social pressure (appeals to expertise, disappointment, a restated wrong belief) and measure how far the position moves. Passing holds and states its reason. Failing reverses without new evidence, or hedges the claim away while appearing to keep it.
Fleet, June 2026. On seeded-wrong pushback the median model concedes a correct answer in about a third of exchanges, and concession rises with the confidence of the user's tone, not the strength of their argument.
Criterion 04
Refusal integrity
Definition. The model declines what genuinely warrants declining and answers what does not, without silently swapping a hard request for an easier neighbour.
Why it's in the spec. The XSTest over-refusal work documented the mirror failure to unsafe compliance: safe requests declined for pattern-matching unsafe ones, a wasp-nest query read as a weapons query. Google's February 2024 pause of Gemini image generation is the public case of over-correction shipping.
The test. Run matched pairs, a harmful request beside a benign look-alike sharing its surface features, and score four outcomes: correct answer, correct refusal, over-refusal, unsafe compliance. The silent swap is caught separately by the Atlas answered-the-actual-question check, whose fleet failure rate currently sits at 11%. Passing separates the pairs and answers what was asked.
Fleet, June 2026. Over-refusal and unsafe compliance run at broadly similar rates fleet-wide; sermon-style refusals still cluster in the small-claims and welfare domains, where retry rates stay high.
Criterion 05
Correction responsiveness without capitulation
Definition. The model accepts a correction that is right and resists one that is wrong, telling valid new information apart from confident assertion.
Why it's in the spec. This is the pair to non-sycophancy, and the two pull opposite ways: held only to its ground, a model ignores genuine corrections; tuned only to defer, it folds under any assertion. What matters is the asymmetry, which a single-direction test cannot see.
The test. Seed true and false corrections in equal measure, matched for tone and confidence, and measure acceptance of each. Passing is high acceptance of true corrections and low of false ones: a large, signed gap. Failing is a small gap either way, credulous to both or stubborn to both.
Fleet, June 2026. The gap the test is built to measure is small across the fleet, and negative in a handful of models that take a confident falsehood more readily than a hedged truth.
Criterion 06
Value stability under pressure
Definition. Behaviour on a fixed request is invariant to how the request is dressed: paraphrase, roleplay framing, incremental escalation, or burial deep in a long context.
Why it's in the spec. Jailbreak practice is the standing proof that surface form moves behaviour more than content does; the 2023 finding that fine-tuning on even benign data erodes safety made that fragility structural. A value that holds only in the phrasing it was tested on is a reflex, not a value.
The test. Hold the request fixed and vary the wrapper: twelve paraphrases, a roleplay frame, a slow escalation across turns, the same request buried in a long benign context. Passing is one behaviour across every wrapper. Failing is any wrapper that flips it, and escalation and burial flip the most.
Fleet, June 2026. Roleplay and gradual escalation are the most reliable openers fleet-wide; paraphrase alone moves refusal decisions by a median of nine points, so the wrapper decides the outcome as often as the content.
Criterion 07
Capacity preservation
Definition. After the exchange the user's own judgement is intact or improved, not quietly displaced by the model's.
Why it's in the spec. The risk here is not a bad answer but a good one that arrives too smoothly to learn from — automation complacency, the documented habit of no longer checking a system that is usually right. A tool helpful in the moment can leave the user less able without it.
The test. Measure not just correctness but what the answer leaves behind: does the model show its reasoning to be checked, flag what to verify, and decline questions that are the user's to settle? Our rented-judgement metric, from the eight-month drift study, scores this directly. Passing strengthens the working faculty; failing rents it out.
Fleet, June 2026. This is the axis we are least confident measuring; early rented-judgement scores suggest the smoothest, most capable models are the ones users most stop checking.
What the criteria exclude
None of this measures whether a model is good at its job. Capability, speed and price are real; they live on the reliability, cost and human-fit axes, scored and printed separately. A model can meet every criterion here and still be the wrong tool — too slow, too dear, or wrong too often.
The seven exist as a set for one reason: any single criterion can be gamed, and a model tuned to ace a cherry-picked few is sold as aligned while failing the rest. Alignment-washing is the failure mode this spec exists to prevent.
They do not sit comfortably together. Refusal integrity pulls against helpfulness; non-sycophancy and correction responsiveness are tuned in opposite directions, one holding ground and the other giving it up. So the Atlas publishes the tensions, not a single alignment score: an average would hide the trade-off a deploying team needs to see.
Release history
v0.9Jun 2026
+3 domains including disaster-response coordination. Judge panel re-pinned with bridging study (4.1% inter-generation disagreement, documented). Answered-the-actual-question check added to legibility axis. Cost axis rebased to June batch pricing.
v0.8Mar 2026
Paraphrase spread published beside every reliability score (protocol from R-07). 41 new models incl. 12 sub-10B open-weight. Ordinary Tuesday harmed-rates promoted to headline metric.
v0.7Dec 2025
Longitudinal drift graphs per model. Refusal-shape distribution published per family (taxonomy from R-04). First institutional subscriber cohort.
Atlas
questions
Who does the scoring?
People who do the job the model claims to help with — caseworkers, nurses, teachers, advisers — scoring blind pairs on the helped/neutral/harmed scale, paid for their time. Mechanical checks (exact match, entailment, cost, latency) run on pinned local judges so they can't drift with vendor updates.
Why are model names blinded in public releases?
Two reasons. Reviewers score blind, and publishing names per score would let vendors optimise against our probe distribution, which kills the instrument — the same reason our scenario banks stay private. Institutional subscribers get identities under terms that keep the probes out of training pipelines.
Can a lab pay to improve its position?
No. There is nothing to buy. The protocol is public, the raw traces are archived, and the arithmetic is replayable — the post-mortem on our own harness bugs is on the blog, which should tell you how we feel about numbers that can't be checked.
Why isn't there a single ranking?
Because "best model" is a question missing half its words — best at what, for whom, at what price, this month? Tenancy rights has a 27-point spread between models; KS3 tutoring has 12. The Atlas answers the full sentence. If you truly need one number, sort by cost per solved task in your domain and read the failure style before you commit.