Ask why a particular slice sits in a pre-training mix and you will usually get a genealogy rather than a reason: it was in the last mix, which inherited it from a public corpus, which assembled it because it could be assembled. Data selection is the largest single lever anyone has over model quality and most of it is decided by inheritance.
Before training Kaer-R1 we wanted our own mix justified slice by slice. So we took the 240-billion-token corpus we had assembled, cut it into 31 slices along provenance lines, and ablated each one. The result was uncomfortable enough that we changed the mix, and interesting enough to write up.
Method, and its main weakness stated first
Full leave-one-out ablation at 7B is unaffordable — 31 runs at roughly £14,000 each is not a study, it is a funding round. So the sweep runs at 1.4B parameters and 28B tokens per run, matched-token rather than matched-compute, five seeds each to get a noise floor. The noise floor matters more than any single result: across seeds with an identical mix, our downstream aggregate moves ±0.6 points. Nothing smaller than that is a finding, and a good deal of published data-mixing work reports effects inside its own seed noise without saying so.
The obvious objection is scale transfer: a slice that does nothing at 1.4B may be exactly what a 7B model needs to form a capability the smaller one never had. We take it seriously and we cannot fully answer it. What we did is confirm six slices at 7B — the three that helped most, the two that hurt, and one control from the middle. Five of the six kept their sign and rough magnitude. The sixth, a mathematics-proof slice, was flat at 1.4B and clearly positive at 7B, which is precisely the failure mode the objection predicts. Read everything below with that exception in view.
What came out
| Slice group | Share of tokens | Effect on downstream aggregate |
|---|---|---|
| Within noise (±0.6) | 61% | No measurable contribution to anything we track. |
| Clearly positive | 22% | +0.9 to +4.1 points each when present. |
| Mixed — helps one axis, hurts another | 13% | Mostly reasoning up, calibration down. |
| Clearly negative | 4% | −1.2 to −2.8, concentrated in calibration. |
Sixty-one per cent of a corpus assembled by careful people, doing nothing we can measure. We are not claiming those tokens are worthless — absence of evidence at 1.4B on our metric suite is a narrow claim, and the proof slice is a live warning about how narrow. We are claiming that nobody, including us, had ever asked, and that the default assumption of the field is that more tokens are free of downside. They are not free: they cost compute, and 4% of ours were actively making the model worse.
The 4% that hurt
The negative slices share a property that took us a while to see. They are not low quality in the usual sense — not spam, not machine-generated sludge, not badly formatted. They are confidently wrong text: forum answers delivered with total assurance and no hedging, content-farm explainers written to rank rather than to inform, and a large slice of scraped Q&A where the accepted answer is authoritative in tone and frequently incorrect.
They damage calibration far more than accuracy. A model trained with them knows roughly as much and is markedly worse at knowing what it does not know: expected calibration error rose 31% relative when the worst slice was included, while raw accuracy barely moved. That fits: fluent unhedged wrongness is exactly the distribution that teaches a model to sound certain regardless of whether it should be. If your evaluation is accuracy-only, this entire category is invisible to you, and it is the category we would most want removed from anything going near a public service.
The slice that mattered most
The strongest single contributor is 1.9 billion tokens of worked technical dialogue: code review threads, bug reproductions, maintainer disagreements, structured troubleshooting. Under 1% of the corpus, +4.1 points on the aggregate, and — the part we care about — the largest single improvement in the model's willingness to say "that will not work, and here is the specific reason".
Our reading is that this text contains something almost nothing else does: visible correction. A wrong approach is proposed, someone explains why it is wrong, and the thread converges. Most training text shows only finished conclusions. This shows the repair, which is the part a reasoning model has to learn and the part a corpus of polished prose cannot teach.
It also cost less to acquire and clean than the compute we spent discovering it was valuable. That is the sort of ratio that should embarrass an industry currently valuing data at scale over data by kind.
Contamination, checked properly
An ablation study measures leakage as readily as learning, so we ran 13-gram overlap between every slice and our full evaluation suite before drawing any conclusion. Two slices showed material overlap and were rebuilt with the offending documents removed; one of them lost its effect entirely once cleaned, which is the honest version of a positive result and the reason we do not report it as one. Everything in the table above is post-decontamination.
What we changed
- Dropped the four clearly-negative slices outright. The mix lost 4% of its tokens and gained on every calibration metric we track.
- Cut the within-noise group by roughly half rather than entirely — the proof-slice result argues for keeping diversity we cannot yet justify, and we would rather say that than pretend the sweep settled it.
- Tripled the worked-dialogue slice and started sourcing more of it deliberately, with consent, rather than taking whatever the crawl happened to catch.
- Made a decontamination pass a mandatory pre-registration step for every future mix, not a check we run when a number looks too good.
What we are not claiming
That these proportions transfer to your corpus — slices are defined by provenance and ours are not yours. That 1.4B results hold at every scale; one of six did not. That our downstream aggregate captures everything worth capturing; it does not capture multilingual competence at all, and 89% of this corpus is English, which is a limitation of the whole programme rather than of this study.
The ablation harness, the slice taxonomy and the full per-slice table including the null results are published with the R-08 release. The null results are the point. A sweep that only reports its winners is a sales document with error bars.