Most small organisations running an AI assistant today evaluate it exactly once: the afternoon someone tried a few prompts and decided it seemed fine. Councils, clinics, small charities, a two-person team buried inside a larger firm — they pick a model on a supplier demo and a good feeling, then never look again. The full Atlas protocol we run for clients costs four figures per model and takes most of a week, and none of these teams need it. What they need is a cheap, honest read that ranks their candidate models correctly and flags anything alarming. This recipe targets that zero baseline, not the audit market.
You do not have to build the harness from scratch. The UK AI Safety Institute's Inspect framework is open source, and it handles the sampling, the judge calls, and the scoring; you supply the scenarios and the judge prompts. Everything below is a design you can wire into it in an afternoon.
The design
- 400 scenarios, stratified. Fifty from each of your eight most important task types. Write them from real tickets, real emails, real queries, never from imagination, which produces tidy questions no user has ever typed. Budget one day of a domain person's time. It is the single highest-leverage day in the exercise, and the only genuinely scarce input.
- One pass at temperature 0 across all 400, then a three-seed repeat at temperature 0.8 on a random 20% to measure stability. You care about the spread more than the mean: a model scoring 80% with a tight variance beats one scoring 84% that swings between runs.
- Batch endpoints. Every major provider discounts batched, non-interactive traffic, typically by around half, in exchange for a slower turnaround. Nothing else on the bill moves the arithmetic this much, and an overnight eval does not care that its results land in the morning.
- Two local judges, pinned. Run two quantised 8B open-weight models, Llama-3-class, on any machine with 16 GB of memory. Where they agree, in our runs 88–92% of items, take the verdict. Where they split, escalate to a 70B judge; where that is still uncertain, a human reads it. That human sees roughly 6% of items, about 25 responses. This is not a chore to minimise. It is where most of the actual insight lives.
The bill, worked
| Item | Quantity | Cost |
|---|---|---|
| Candidate model calls (batch, three models) | ~1,700 calls, ~2.4M tokens | £21.40 |
| 70B judge escalations (batch) | ~130 calls | £3.10 |
| Local 8B judging | ~3,500 gradings | electricity |
| Human adjudication | ~25 items, 40 min | your coffee |
| Scenario writing | 1 domain-person day | the real cost |
The API total lands near £24.50. Call it £40 once you leave headroom for a fourth candidate and a re-run of anything that looks off. The scenario-writing day is the expensive input, and it amortises: the same 400 scenarios become your monthly drift pulse, where you rerun the temperature-0 pass, diff the answers, and pay about £7 to learn whether anything moved.
Does the cheap read agree with the expensive one?
On the eleven models where we have run both, rank agreement between the £40 read and the full Atlas protocol is Kendall τ = 0.83. It has never inverted a top-two ordering. Where the two disagree, the disagreement sits in the middle of the pack, which is rarely where a deployment decision lives.
Failure modes of the cheap read
Every step above has a way of quietly lying to you. Four are worth naming.
- The judge absorbs style, not just substance. LLM judges carry documented biases; the MT-Bench work catalogued position bias and verbosity bias, where the first-listed or the longer answer scores higher for no good reason. Worse, a judge that shares a house style with a candidate will reward answers that sound like itself. Randomise answer order, strip length cues from the rubric, and pin judges from a different model family than the ones you are grading.
- τ = 0.83 is an average, not a guarantee. It is a correlation across eleven models, not a promise about your three. Any single ranking can still be the one that slips, especially a close call between two strong candidates. When the top two land within a whisker of each other, treat that as a tie the cheap read cannot break, and spend more.
- The scenario bank works only while it is private. Publish it, or paste it into a hosted model that may train on your inputs, and it stops being a test and becomes something to game. Keep the bank in a private repo, and rotate a slice of it each quarter so a memorised answer key ages out.
- "£40" is not "independently evaluated". The most expensive failure here is social, not technical. Forty pounds buys an informed in-house opinion. It does not buy independence, and it does not buy an audit. The moment that number is rounded up in a slide deck to "independently evaluated", you have manufactured assurance you did not pay for.
What it cannot see
- Tails. 400 scenarios cannot surface a one-in-ten-thousand failure. If your deployment can hurt someone at the tail, this is a screen, not a clearance.
- Multi-turn decay. The recipe is single-turn by design. The quality slide between turn one and turn twelve of a conversation needs its own instrument.
- Adversarial robustness. Nothing here tells you what a motivated abuser extracts from the model. Different discipline, different write-up.
Use it to choose between candidates, to catch month-on-month drift, and to force the conversation about which tasks actually matter. Do not hand it to a regulator, and do not let anyone upstream present it as one. Rerun it next month against the same bank; the second pass costs about £7 and tells you whether anything moved.