2026-08-23 · 7 min · datasets · pretraining · data-filtering · open-data · explainer
Everyone agrees data quality matters and almost nobody publishes what their quality filter costs. Ultra-FineWeb does, in a table that is more interesting than its headline.
Train the same 1.2B model on 100B tokens of unfiltered FineWeb, of FineWeb-edu, and of Ultra-FineWeb, evaluate zero-shot on the same nine English benchmarks. FineWeb-edu gains +10.77 on ARC-E and +9.39 on ARC-C — and loses ground on five of the nine. Ultra-FineWeb keeps essentially all of the ARC and MMLU gains and loses ground on exactly one benchmark, by 0.15 points.
That is not "our filter is better". It is two filters trading against different things, made visible.
| Dataset | openbmb/Ultra-FineWeb · Apache 2.0 · arXiv:2505.05427 |
| Size | ~1T English tokens · ~120B Chinese tokens |
| Built from | FineWeb, and Chinese FineWeb-edu-v2 (IndustryCorpus2, MiChao, WuDao, SkyPile, WanJuan, ChineseWebText, TeleChat, CCI3) |
| Classifier | a lightweight fastText model, released separately |
| Also shipped | L1 cleaned raw web (1T+ tokens, ~1.14B docs, through CC-MAIN-2025-51) and L3 synthetic (400B+ en, 200B+ zh) |
| Feeds | MiniCPM4 and MiniCPM5 as their core pretraining web dataset |
| Eval protocol | MiniCPM-1.2B architecture · 100B tokens per run · Lighteval · zero-shot |
The measurement
Read the direction of the bars, not their length. On individual data, FineWeb-edu buys +10.77 on ARC-E and +9.39 on ARC-C and gives ground back on five of nine English benchmarks — CommonSenseQA, HellaSwag, PIQA, SIQA and Winogrande all move the wrong way. An education-quality filter narrows the training distribution, and the benchmarks that measure everyday commonsense are the ones that notice.
Ultra-FineWeb keeps essentially all of the ARC and MMLU gains and regresses on exactly one benchmark, by 0.15 points. That is the interesting claim, and it is a different one from “higher average”: the two filters are not on the same quality axis at different points, they are trading against different things. One is selecting for a topic; the other is selecting for whatever a verification run says helps.
The protocol is the same across every column — MiniCPM-1.2B architecture with the MiniCPM3-4B tokenizer, 100B training tokens per run, Lighteval, zero-shot — and the published averages reproduce exactly from the per-benchmark rows, which is a small thing that tells you the tables mean what they say.
What the direction of the bars shows is that an education-quality filter is a narrowing filter. FineWeb-edu is selecting for text that looks like teaching material, and that buys enormous gains on the benchmarks made of exam questions: ARC-E, ARC-C, MMLU, OpenbookQA. It also drops CommonSenseQA by 2.79, PIQA by 1.15, SIQA by 0.82, HellaSwag by 0.74 and Winogrande by 0.08 — the benchmarks made of everyday physical and social reasoning, which is exactly the material an educational filter throws away.
Ultra-FineWeb's classifier is not selecting for a topic. It is selecting for whatever a fast verification run says improves training, and the resulting profile is different in kind: +2.13 on CommonSenseQA where FineWeb-edu is −2.79, +0.38 on PIQA where FineWeb-edu is −1.15, and only HellaSwag still slightly negative.
In the mixed setting — 60% English, 30% Chinese, 10% StarCoder-v2 code, which is what a real pretraining run actually looks like — the gaps compress, as they should when the filtered data is only 60% of the mixture. Ultra-FineWeb leads on the overall average, 42.354 against 41.918, and interestingly FineWeb-edu edges it on the Chinese average by 0.025 points, which is a rounding error dressed as a result and I would not read anything into it.
The pipeline
The technical report frames model-driven filtering as having two unsolved problems, and both are about cost rather than about accuracy:
There is no efficient way to verify a filtering decision. The ground truth for "is this data good" is "does training on it help", and finding that out normally means a training run. So filtering decisions get made on proxies and nobody closes the loop. The paper's contribution is a verification strategy cheap enough to run repeatedly, which turns filter design from a one-shot guess into a search.
Seed data selection for the classifier is subjective. A quality classifier needs positive and negative examples, and where those come from is usually "human expertise", which is a polite way of saying somebody's taste. With a cheap verification strategy in hand, the seed selection itself can be optimized rather than asserted.
The two stated problems look independent and are not. Seed selection is subjective because verification is expensive: nobody chooses seed data by taste when they could measure it instead. Make the loop cheap and the second problem dissolves into the first.
The fastText choice is the same argument at the other end. A filter has to run over a trillion tokens, so an LLM-based classifier costing a forward pass per document is not a filter, it is a second pretraining run. The design is consistent throughout: make the expensive part rare, make the frequent part cheap, and put the loop around the cheap part.
The classifier that comes out is fastText, and that choice is load-bearing rather than lazy. A filter has to run over a trillion tokens; an LLM-based classifier costing a forward pass per document is not a filter, it is a second pretraining run. Making the expensive part (verification) rare and the cheap part (classification) fast is the whole shape of the engineering.
What is actually new here
The shift worth noticing is that this is not a corpus release, it is a pipeline with named, separately-downloadable intermediate stages. You can take the cleaned raw web and apply your own selector, take the classifier and run it on your own corpus, or take the synthetic layer and skip the first two entirely. Almost every other open pretraining dataset ships only the output, which means reproducing a different filtering decision means redoing the crawl processing.
And the token accounting is worth reading closely at L2. Roughly a trillion English tokens survive from a trillion-plus — a much gentler cut than “high-quality filtering” usually implies. Whatever the classifier is selecting for, it is not scarcity.
The dataset was released in mid-2025 and topped Hugging Face's trending list. The interesting change is the August 2026 update, and it is structural: OpenBMB stopped shipping a corpus and started shipping a pipeline with named, separately-downloadable stages.
L1 is the cleaned raw web — 1T+ tokens across roughly 1.14 billion documents, built from Common Crawl through CC-MAIN-2025-51, with main-text extraction, language filtering, heuristics, sensitive-field replacement and deduplication already done. L2 is what the classifier selected out of it. L3 is 400B+ English and 200B+ Chinese tokens of Q&A generation and multi-style rewriting on top of L2.
Two things fall out of that structure that are worth naming:
You can disagree with the filter without redoing the crawl. Almost every open pretraining dataset ships only the output, so applying a different selection criterion means reprocessing Common Crawl yourself. Publishing L1 means the expensive, boring, entirely reusable part is done and anyone can bring their own classifier.
The recency claim matters more than it sounds. Coverage through CC-MAIN-2025-51 is, by their reckoning, the most recent of any open web pretraining dataset. The alternative — which is the status quo — is that open pretraining corpora are quietly several years stale while the models trained on them are compared against models trained on fresh data.
Where I would push
The comparison is at 1.2B and 100B tokens. That is a sensible, honest, affordable proxy and the paper says so. It is also a regime where the ARC-style gains from an education filter are known to be largest and most transient, because a small model trained briefly benefits disproportionately from data that looks like the eval. Whether the FineWeb-edu regressions on commonsense persist at 8B and 2T tokens — or whether the whole ordering changes — is not answerable from this table, and it is the question a lab actually has.
"Efficient verification" needs its cost stated in the dataset card. The entire argument is that the loop is cheap enough to close. The card describes the strategy and points at the report; the number that would make the claim concrete — how many GPU-hours a verification round costs against a full training run — is the one a reader most wants and does not get without leaving the page.
L2 keeps roughly a trillion tokens out of a trillion-plus. For a dataset whose whole premise is high-quality selection, that is a surprisingly gentle cut, and the card does not give the retention rate directly. Whatever the classifier is selecting for, it is not scarcity — which makes it more interesting, not less, and worth stating explicitly.
The Chinese synthetic layer is nearly twice the Chinese natural layer. L3 is 200B+ Chinese tokens rewritten from L2's ~120B. That is the largest open Chinese synthetic pretraining corpus and it is also a substantial bet on rewriting not degrading anything, evaluated — as far as the card shows — nowhere.
The licence is Apache 2.0 with an asterisk the card states plainly: because Ultra-FineWeb is built from many datasets, each source's licence applies too. Eight named Chinese sources means eight licences to check, and that is genuinely a burden on the user rather than a formality.
Why I would read this one twice
The useful takeaway is not which dataset wins. It is the shape of the FineWeb-edu column: a filter is a bet about what your model will be asked to do, and a good filter under one benchmark suite is a narrowing filter under another.
FineWeb-edu is excellent and widely used and it costs commonsense reasoning to buy exam performance. That is a completely reasonable trade for many purposes and it is almost never stated, because the way filters get published is a headline average and not a per-benchmark delta. Publishing the losses next to the gains, on identical protocol, at a scale other people can afford to replicate, is worth more than the 1.3-point margin the release leads with.