We are on our third evaluation harness in eighteen months. Neither of the first two died of anything clever. They died of mistakes you would catch in review at any competent data shop, and we walked past both because we treated the harness as a script we ran, not the system of record it was.
Start from something the field already knows: the ruler is contested before you pick it up. The same open model, run through different harnesses on the same benchmark, returns different scores, because prompt formatting, answer parsing and log-likelihood-versus-generation scoring are all choices made differently by everyone. Hugging Face once had to explain in public why a model scored worse than expected on MMLU: several implementations of the same test were in circulation, each giving its own number. HELM was Stanford's attempt to pull all this onto common ground; it helped, and it did not settle the argument.
Harness one: the cache that lied for six weeks
V1 was nine hundred lines of async Python with a response cache bolted on to save money on repeated runs. The key was (model_id, prompt_hash). That is the entire bug. It should have keyed on the full request envelope: temperature, top_p, max_tokens, the system prompt, the API version. If a field changes the distribution of outputs it belongs in the key, and sampling parameters change everything.
For six weeks a run configured at temperature 0.7 was quietly served a cached temperature-0 response, and the reverse, and we scored all of it. The variance figures for that window were fiction. Nothing threw an error. What gave it away was that some models looked impossibly stable from run to run, and reproducibility that clean is not a triumph, it is a symptom. We deleted six weeks of results and shipped that quarter's Atlas release late, with a note saying why.
Harness two: the truncation that flattered concision
V2 was a genuine rewrite: per-provider adapters, a proper queue, streaming. The streaming client is where it got us. Under a particular timeout race on one provider it returned responses that had stopped early but looked complete: valid JSON, no error flag, the text simply ended. Nothing downstream could tell.
Truncated answers are shorter, and our rubric gave marks for concision. So for about three weeks one provider's models climbed on communication efficiency while we scored the first sixty per cent of their replies as if that were the whole thing. The scores did not catch it; a length-distribution monitor did, one added for capacity planning and nothing to do with quality. It had grown a second hump. A bimodal histogram where there should be one mode is the system telling you two things are being counted as one.
Harness three: boring, append-only, replayable
V3 is architecturally dull, which we now understand to be the point. Every session is an immutable JSONL record: full request envelope, raw response bytes, a content hash of those bytes, timings, cost. Metrics are pure functions over that store, each stamped with the version hash of the code that produced it. Change a rubric and you can rescore two years of history in an afternoon, which we have done twice, because the inputs never move. Configs are data too: a run is a YAML file whose hash lands in every record it touches, so "what produced this number" is a lookup, not an archaeology project.
Keeping every raw byte looks like paranoia until someone else becomes entitled to ask what your system said. In February 2024 the British Columbia Civil Resolution Tribunal held Air Canada liable for wrong information its own chatbot gave a passenger, rejecting the idea that the bot was a party the airline need not answer for. Moffatt v. Air Canada is a small-claims ruling, but the line is clean: if your system said it, you own it, and the exact bytes it emitted are what someone will read back to you. Dedupe was the unplanned bonus: content hashing cut provider spend by roughly thirty-four per cent, because identical probes across releases are paid for once.
The two things that move underneath you
A perfect harness still measures a moving target with a moving instrument, and either wrecks a longitudinal comparison.
The subject moves. In December 2023 enough people reported that GPT-4 had turned "lazy", terser and quicker to refuse, that OpenAI acknowledged it in public and noted the model had not been updated in weeks. Track a hosted model for months and some of your trend line is the provider's weather, not your signal. Deprecation is worse: the provider retires the snapshot you were following, the endpoint routes to its successor, and your before-and-after straddles two systems sharing a name. Record the exact snapshot on every call and assume it will be taken away.
The instrument moves too, which is the wart we have not fixed. We score much open-ended work with judge models, and judges update like anything else. We pin local ones (quantised 8B and 70B models on our own boxes) so their behaviour holds, but the day we rotate them every comparison spanning the change needs a bridging study: double-score a sample under both generations and measure the offset. Our first bridge, in April, put inter-generation disagreement at 4.1 per cent, almost all of it on borderline refusals, where a soft partial answer sits between compliance and refusal. Manageable, but a standing tax I have not seen anyone else in eval work cost out loud.
The two dead harnesses cost roughly £31,000 in rerun compute and lost calendar time, plus a quarter of the credibility you only notice once you have spent it. The list below is what that bought.
If you're building a harness this quarter
- Key the cache on the whole request envelope; a field that moves the output distribution moves the key.
- Store raw response bytes, hashed at write time; score the archive, never re-streamed text.
- Alert on variance that drops, not only variance that spikes; reproducibility that looks perfect is a bug report.
- Watch response-length distribution per provider and model; a second hump means two things scored as one.
- Make configs data and stamp the config hash into every record, so provenance is a lookup.
- Record the exact model snapshot per call, and expect deprecation to split your time series in two.
- Pin your judge models; budget a bridging study before each rotation, because the instrument drifts too.
- Keep the archive complete enough to hand a lawyer; Moffatt v. Air Canada means someone may ask what you said.