Research
What we've measured while building and evaluating advanced systems, written up with the methods attached. Numbered entries are studies; most link to a full write-up on the blog, the rest are partner reports available on request.
Paraphrase sensitivity across 312 models: a 19-point median accuracy spread across twelve phrasing families, against an 11-point gap between adjacent-tier models. Question-order dominates; typos barely register.
Longitudinal trace of one deployed assistant: nine behavioural step-changes (six unannounced), flat task accuracy, monotonic style drift, and one month where date arithmetic quietly broke and recovered.
A minimal decision-grade evaluation recipe: 400 stratified scenarios, pinned local judges, batch pricing. Kendall τ = 0.83 rank agreement with the full Atlas protocol at roughly 1% of the cost.
Hand-coding of 9,400 refusals yields a six-shape taxonomy. Silent scope-narrowing — answering an easier nearby question with no signal — is the most common shape in the flagship class and invisible to standard rubrics.
Scoring models on realistic traffic: consented, hand-paraphrased scenarios with domain-worker judges and a helped/neutral/harmed scale. Leaderboard rank explains 41% of helped-rate variance; harmed-rates spread 4–19%.
Pre-deployment and live-traffic evaluation for three London borough services: harmed-rate bounds, deadline-question safeguards, and the human-fallback usage metric now adopted by all three partners. Partner data; summary available on request.
Foundation document for the Model Atlas: domain taxonomy, blind pairing design, judge-panel construction and pinning policy, paraphrase-spread reporting, and the five published axes. Updated with each release.
Programme one
Does it, usually — not can it, ever. Fixed probes, fixed seeds, published variance. The paraphrase study (R-07) and the Ordinary Tuesday protocol (R-03) are this programme's instruments.
Feeds the Atlas reliability and cost axes.
Programme two
An answer you can't trace is a rumour with good grammar. Refusal taxonomy (R-04), answered-the-actual-question checks, and mechanism tracing on open-weight systems.
Feeds the Atlas legibility axis.
Programme three
Behaviour under pressure, over months. The eight-month drift trace (R-06) is the template: frozen probes, change-point detection, manner metrics that move before competence does.
Feeds the Atlas human-fit and drift axes.
Every lab says "aligned with human values" and hopes nobody asks which ones. We keep a list. It's short, unglamorous, and non-negotiable — the working faculties that make a human, human. A system passes when it strengthens them in the person using it. It fails when it quietly rents them out.
A thinker worth building doesn't just produce. It understands what it's producing, for whom, and what it costs them. Our rented-judgement metrics (in R-06) are the first attempt to put numbers on exactly that.
See it measured