# Ultra-FineWeb: what an education filter costs you, measured

> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/ultra-fineweb
> date: 2026-08-23
> tags: datasets, pretraining, data-filtering, open-data, explainer
Everyone agrees data quality matters and almost nobody publishes what their quality filter *costs*. [Ultra-FineWeb](https://huggingface.co/datasets/openbmb/Ultra-FineWeb) does, in a table that is more interesting than its headline.

Train the same 1.2B model on 100B tokens of unfiltered FineWeb, of FineWeb-edu, and of Ultra-FineWeb, evaluate zero-shot on the same nine English benchmarks. FineWeb-edu gains **+10.77 on ARC-E and +9.39 on ARC-C** — and loses ground on **five of the nine**. Ultra-FineWeb keeps essentially all of the ARC and MMLU gains and loses ground on exactly one benchmark, by 0.15 points.

That is not "our filter is better". It is two filters trading against different things, made visible.

| | |
|---|---|
| Dataset | [openbmb/Ultra-FineWeb](https://huggingface.co/datasets/openbmb/Ultra-FineWeb) · Apache 2.0 · [arXiv:2505.05427](https://arxiv.org/abs/2505.05427) |
| Size | **~1T English** tokens · **~120B Chinese** tokens |
| Built from | FineWeb, and Chinese FineWeb-edu-v2 (IndustryCorpus2, MiChao, WuDao, SkyPile, WanJuan, ChineseWebText, TeleChat, CCI3) |
| Classifier | a lightweight **fastText** model, released separately |
| Also shipped | **L1** cleaned raw web (1T+ tokens, ~1.14B docs, through CC-MAIN-2025-51) and **L3** synthetic (400B+ en, 200B+ zh) |
| Feeds | MiniCPM4 and MiniCPM5 as their core pretraining web dataset |
| Eval protocol | MiniCPM-1.2B architecture · 100B tokens per run · Lighteval · zero-shot |

## The measurement

<FilterComparison />

The protocol is the same across every column — MiniCPM-1.2B architecture with the MiniCPM3-4B tokenizer, 100B training tokens per run, Lighteval, zero-shot — and the published averages reproduce exactly from the per-benchmark rows, which is a small thing that tells you the tables mean what they say.

What the direction of the bars shows is that **an education-quality filter is a narrowing filter**. FineWeb-edu is selecting for text that looks like teaching material, and that buys enormous gains on the benchmarks made of exam questions: ARC-E, ARC-C, MMLU, OpenbookQA. It also drops CommonSenseQA by 2.79, PIQA by 1.15, SIQA by 0.82, HellaSwag by 0.74 and Winogrande by 0.08 — the benchmarks made of everyday physical and social reasoning, which is exactly the material an educational filter throws away.

Ultra-FineWeb's classifier is not selecting for a topic. It is selecting for whatever a fast verification run says improves training, and the resulting profile is different in kind: +2.13 on CommonSenseQA where FineWeb-edu is −2.79, +0.38 on PIQA where FineWeb-edu is −1.15, and only HellaSwag still slightly negative.

In the mixed setting — 60% English, 30% Chinese, 10% StarCoder-v2 code, which is what a real pretraining run actually looks like — the gaps compress, as they should when the filtered data is only 60% of the mixture. Ultra-FineWeb leads on the overall average, 42.354 against 41.918, and interestingly FineWeb-edu edges it on the Chinese average by 0.025 points, which is a rounding error dressed as a result and I would not read anything into it.

## The pipeline

<Figure
  src="/articles/ultra-fineweb/fig1.png"
  alt="A left-to-right pipeline diagram of the Ultra-FineWeb data filtering process, showing seed data selection feeding classifier training, an efficient verification stage that evaluates candidate data's effect on model training at low cost, and the resulting lightweight fastText classifier applied over the FineWeb and Chinese FineWeb corpora to produce the filtered Ultra-FineWeb dataset."
  caption="The two problems the pipeline is designed around: verifying data quality cheaply, and choosing seed data for the classifier without relying on human judgement. (OpenBMB, Ultra-FineWeb.)"
/>

The technical report frames model-driven filtering as having two unsolved problems, and both are about *cost* rather than about accuracy:

**There is no efficient way to verify a filtering decision.** The ground truth for "is this data good" is "does training on it help", and finding that out normally means a training run. So filtering decisions get made on proxies and nobody closes the loop. The paper's contribution is a verification strategy cheap enough to run repeatedly, which turns filter design from a one-shot guess into a search.

**Seed data selection for the classifier is subjective.** A quality classifier needs positive and negative examples, and where those come from is usually "human expertise", which is a polite way of saying somebody's taste. With a cheap verification strategy in hand, the seed selection itself can be optimized rather than asserted.

<VerificationLoop />

The classifier that comes out is **fastText**, and that choice is load-bearing rather than lazy. A filter has to run over a trillion tokens; an LLM-based classifier costing a forward pass per document is not a filter, it is a second pretraining run. Making the expensive part (verification) rare and the cheap part (classification) fast is the whole shape of the engineering.

## What is actually new here

<DataTiers />

The dataset was released in mid-2025 and topped Hugging Face's trending list. The interesting change is the August 2026 update, and it is structural: OpenBMB stopped shipping *a corpus* and started shipping a **pipeline with named, separately-downloadable stages**.

L1 is the cleaned raw web — 1T+ tokens across roughly 1.14 billion documents, built from Common Crawl through `CC-MAIN-2025-51`, with main-text extraction, language filtering, heuristics, sensitive-field replacement and deduplication already done. L2 is what the classifier selected out of it. L3 is 400B+ English and 200B+ Chinese tokens of Q&A generation and multi-style rewriting on top of L2.

Two things fall out of that structure that are worth naming:

**You can disagree with the filter without redoing the crawl.** Almost every open pretraining dataset ships only the output, so applying a different selection criterion means reprocessing Common Crawl yourself. Publishing L1 means the expensive, boring, entirely reusable part is done and anyone can bring their own classifier.

**The recency claim matters more than it sounds.** Coverage through `CC-MAIN-2025-51` is, by their reckoning, the most recent of any open web pretraining dataset. The alternative — which is the status quo — is that open pretraining corpora are quietly several years stale while the models trained on them are compared against models trained on fresh data.

## Where I would push

**The comparison is at 1.2B and 100B tokens.** That is a sensible, honest, affordable proxy and the paper says so. It is also a regime where the ARC-style gains from an education filter are known to be largest and most transient, because a small model trained briefly benefits disproportionately from data that looks like the eval. Whether the FineWeb-edu regressions on commonsense persist at 8B and 2T tokens — or whether the whole ordering changes — is not answerable from this table, and it is the question a lab actually has.

**"Efficient verification" needs its cost stated in the dataset card.** The entire argument is that the loop is cheap enough to close. The card describes the strategy and points at the report; the number that would make the claim concrete — how many GPU-hours a verification round costs against a full training run — is the one a reader most wants and does not get without leaving the page.

**L2 keeps roughly a trillion tokens out of a trillion-plus.** For a dataset whose whole premise is high-quality selection, that is a surprisingly gentle cut, and the card does not give the retention rate directly. Whatever the classifier is selecting for, it is not scarcity — which makes it more interesting, not less, and worth stating explicitly.

**The Chinese synthetic layer is nearly twice the Chinese natural layer.** L3 is 200B+ Chinese tokens rewritten from L2's ~120B. That is the largest open Chinese synthetic pretraining corpus and it is also a substantial bet on rewriting not degrading anything, evaluated — as far as the card shows — nowhere.

**The licence is Apache 2.0 with an asterisk the card states plainly:** because Ultra-FineWeb is built from many datasets, each source's licence applies too. Eight named Chinese sources means eight licences to check, and that is genuinely a burden on the user rather than a formality.

## Why I would read this one twice

The useful takeaway is not which dataset wins. It is the shape of the FineWeb-edu column: **a filter is a bet about what your model will be asked to do**, and a good filter under one benchmark suite is a narrowing filter under another.

FineWeb-edu is excellent and widely used and it costs commonsense reasoning to buy exam performance. That is a completely reasonable trade for many purposes and it is almost never stated, because the way filters get published is a headline average and not a per-benchmark delta. Publishing the losses next to the gains, on identical protocol, at a scale other people can afford to replicate, is worth more than the 1.3-point margin the release leads with.