The Kaer Labs blog
Written up, worked out
Findings with their methods attached, engineering post-mortems with the bills included, and the occasional argument. Everything here was measured before it was written.
Twelve ways to ask the same question
We rephrased 40 questions twelve ways each, ran them across 312 models at temperature zero, and found wording moved accuracy more than changing model did.
Read the study
Three harnesses later: everything our eval infrastructure got wrong
Two rewrites, a six-week contamination we shipped without noticing, and the dull append-only architecture that finally held: a post-mortem for anyone wiring up evaluation infrastructure now.
Alignment is a verb
A model is certified once, then quietly reshaped by a dozen uncoordinated hands, which is why alignment is not a certificate you earn but maintenance you budget for.
Eight months of asking one assistant the same 1,850 questions
For eight months we put the same 1,850 questions to one public-service assistant; competence barely moved, but its manner drifted and it changed nine times, six unannounced.
Cheap intelligence changes who gets to ask
A welfare-navigation pilot let us watch the price of a competent answer fall to three tenths of a penny, and log who began asking once asking cost nothing.
The £40 behavioural read
A worked, honest recipe for a decision-grade read on a model's behaviour for about forty pounds: the sampling, the local judges, the batch pricing, and what it cannot see.
The Ordinary Tuesday protocol
Most benchmarks test the questions a model can be graded on; we built one for the messages people actually send, and scored what happens when it gets them wrong.
Refusals have a grammar
Refusal rates get benchmarked everywhere and refusal styles almost nowhere, so we hand-coded 9,400 declines into six shapes and found the most dangerous one invisible to the user.
One finding a fortnight, by email
No roundups, no "in this issue". One measured thing, explained properly, every other Tuesday.