UltraData-Code: counting a 1.2 TB corpus without downloading it

A dataset is the most checkable thing anyone ships. A model card can claim a benchmark I have no cluster to reproduce; a corpus is a pile of files with a row count, and the row count is either right or it isn't. OpenBMB released openbmb/UltraData-Code on 7 September 2026 with two headline numbers — "UltraData-Code-L2 (400B tokens) and UltraData-Code-L3 (150B tokens) across 11 programming languages" — so I went and counted.

The whole thing is 1,215,994,166,161 bytes of parquet. I have 1.3 GB of free disk. Everything below was measured through Hugging Face's datasets-server APIs and about 11,000 sampled rows, and I say which method produced which number every time.

expand

What got released, exactly

Two configs, eleven splits each, one split per language. The /size endpoint gives exact row counts and byte totals for every one of them, computed by the Hub over the full parquet set rather than sampled:

curl -sS 'https://datasets-server.huggingface.co/size?dataset=openbmb%2FUltraData-Code' \
  | python3 -c 'import json,sys; d=json.load(sys.stdin)["size"];
    print(f"{d[\"dataset\"][\"num_rows\"]:,} rows, {d[\"dataset\"][\"num_bytes_parquet_files\"]:,} bytes")'
# 348,083,481 rows, 1,215,994,166,161 bytes

348,083,481 rows: 266,878,376 in L2 and 81,205,105 in L3. The split is almost exactly even on disk — 611.3 GB for L2, 604.7 GB for L3 — which is the first thing that looks wrong, because L2 is advertised at 400B tokens and L3 at 150B. Nearly equal bytes for a 2.7× token gap means something is being stored more than once.

Counting tokens without the tokens

Row counts are free. Token counts are not, and the card never says which tokenizer produced the 400B and 150B. Here is the method I used, because the method is the number:

1. Exact rows per split from /size, as above.
2. Mean character length per column from /statistics, which returns min, max, mean, median and a histogram for every column. For three splits — L2/r, L3/r, L3/sh — the server reads the whole split and the response carries "partial": false. For the other nineteen it reads the leading shards up to a size cap, between 1.3% and 60% of the rows.
3. Extrapolate total characters as mean × exact rows.
4. Measure characters-per-token by pulling 500 rows per split through /rows (five windows of 100 at offsets 0/20/40/60/80%) and tokenising them with tokenizer.json from openbmb/MiniCPM5-2B — the model this corpus exists to feed. 11,000 rows in total.

Step 2 is the one that should worry you, because a leading-shard sample is not a random sample. So I checked it. /size also returns num_bytes_memory, the Arrow in-memory size of the entire corpus, computed over all 348 million rows — a number my extrapolation has no access to and must nonetheless predict:

predicted  3,059,031,440,460 B     (sum over splits of mean-chars x exact-rows + offsets)
actual     3,061,255,972,282 B     (datasets-server, full corpus)
ratio      0.9993

0.07% off across the corpus, and every individual split within 0.7%. The leading shards are representative. That is the only reason I am willing to publish the next chart.

L2 measures 431.6B tokens against an advertised ~400B. That is 7.9% more than claimed, which is the rare direction. L3's full_content measures 137.1B against ~150B, 8.6% under. Both are inside what a tokenizer swap could plausibly explain, and I would not write a paragraph about either.

The number worth a paragraph is the third bar. content — the column of that name, the one the default config loads, the one the card's own Data Formats section describes as "serialization containing task and solution" — is 52.4B tokens. If you load_dataset L3 and train on content, believing you have the 150B corpus the card advertises, you have about a third of it.

Why L3 is 604 GB

Pull one row and the storage question answers itself. This is UltraData-Code-L3, split py, offset 0, straight off /rows, with the long fields cut:

{
  "uuid": "02ef0d3e-968f-4246-9def-862554b44b2c",
  "content": "Write a Python function `draw_star_square(n)` that takes a positive …",
  "content_format": "markdown",
  "raw_content": "import sys\n\ninput = lambda: sys.stdin.readline().rstrip()\nn = int(input())\n…",
  "task": "Write a Python function `draw_star_square(n)` that takes a positive …",
  "analysis": "The approach is to directly simulate the pattern. First, compute side …",
  "solution": "def draw_star_square(n):\n    \"\"\"\n    Generate a square pattern of …",
  "test": "assert draw_star_square(1) == [\"*\"]\nassert draw_star_square(2) == [ …",
  "full_content": "Write a Python function `draw_star_square(n)` that takes a positive …",
  "full_content_format": "standard order"
}

Measure the fields on that row and the arithmetic is exact: task 746 + analysis 1,645 + solution 1,004 + test 680 = 4,075 bytes, and full_content is 4,078 — the four fields concatenated with three separators. content is 1,766, and task + solution is 1,750. Every generated token is on disk two or three times.

Across the whole of L3 that comes to 402.5B tokens of stored text for a corpus described as 150B — 2.94× redundancy. In bytes of parquet per delivered token, L2 costs 1.416 and L3 costs 4.411 — L3 is 3.1× more expensive to store per token you can actually train on. Plan disk from the card's 150B and you will be short.

raw_content is the other 79.0B tokens, and it is not generated at all — it is the original GitHub source file the exercise was derived from, carried along verbatim. Hold that thought; it comes back in the licence section.

"Algorithmically relevant" is a claim about composition

L2's pitch is that it selects "algorithmically relevant code" out of L1 using a role model and a relevance model. Both models' outputs ship inside the data — category and algo_rel_score are columns — which makes the claim checkable without running anything.

Corpus-wide, weighting each split's /statistics frequency table by its exact row count, L2 is:

The plurality label in "algorithmically relevant code" is TOOL. ALGO is a quarter of it.

The card is not hiding this — it says the framework learns "algorithmic relevance beyond explicit ALGO files", which is exactly what a 26.6% ALGO share looks like. But per language the spread is enormous, and the spread is where the interesting part is. Pick a language:

C++ is 86.7% ALGO and genuinely looks like the thing on the tin. Ruby is 43.1% CONFIG and 10.2% ALGO. PHP is 4.0% ALGO. Shell is 86.3% TOOL. The name "UltraData-Code-L2" describes C++ well and Ruby not at all.

The second panel is the part the release never wrote down anywhere. A corpus cannot hide its own thresholds: the minimum surviving score is the cut. Across all eleven languages no file has a quality_score below 3.0 — the lowest surviving score ranges from 3.00000 in C++ to 3.00019 in JavaScript, which is a >= 3.0 gate seen from the inside. And algo_rel_score has a per-language floor:

cpp 0.85   php 0.85   sh 0.85
js  0.75   rb  0.75
java 0.70
cs  0.60
go  0.50   r   0.50   rust 0.50
py  0.00

That ladder is what "language-adaptive" means in practice, and it is a reasonable thing to do. Nine of the eleven floors sit at a round number. Python's is zero.

Python is not like the other ten

Python is the largest split — 72,844,227 rows, 82.3B tokens — and it is the one the card's headline results table is built on. It is also the only split whose relevance filter effectively isn't one.

In the 1,224,287 rows the statistics server read, 21.76% score below 0.5 and 11.21% score below 0.1. Extrapolated against the exact row count that is roughly 15.9M and 8.2M files. In my 500-row samples every other language bottoms out at its floor in every one of the six categories; Python's lowest ALGO-labelled file scores 3.69e-10, and its lowest non-ALGO file scores 0.103, so the bypass is not the whole story — Python's whole relevance gate is set an order of magnitude lower than anyone else's.

Here is the concrete case, from my sample. Three rows, byte-identical, 5,467 characters each:

row 29,137,716  gokuls999/exam_portal
                venv/Lib/site-packages/pip/_vendor/rich/_ratio.py
row 43,706,618  ChiragJiwnani/DataCraft---End-to-End-Data-Science-Project-Tool
                venv/lib/python3.12/site-packages/pip/_vendor/rich/_ratio.py
row 58,275,388  knagaki1225/memoApp
                venv/lib/python3.12/site-packages/pip/_vendor/rich/_ratio.py

category         ALGO
quality_score    8.447600364685059
algo_rel_score   2.6479629013920203e-05  /  1.3834218407282606e-05

rich/_ratio.py is a genuinely nice piece of algorithmic code — it solves a proportional layout allocation. It is also pip's vendored copy of it, sitting inside three different people's committed virtualenvs. The role model calls it ALGO. The relevance model scores it at 0.000026. Two models that are supposed to agree on the same file, shipped side by side in the same row, disagreeing by five orders of magnitude — and in Python the ALGO label is what carries the file through.

What survived deduplication

L1 is where the card says near-duplicate removal happens: "MinHash signatures are generated and LSH retrieves near-duplicate candidates. Similarity edges define connected components. For groups containing multiple near-duplicates, the highest ranked 50% are retained." L2 is selected from L1, so exact byte-for-byte repeats in L2 are the easiest possible case for that stage — identical files have identical signatures and Jaccard 1.0.

I hashed the content column of all 5,500 sampled L2 rows.

178 of 5,500, or 3.24%, have a byte-identical twin inside the same small sample. PHP is 12.0%, Ruby 10.4%; Java and Rust are zero. And the shape of the duplicates is consistent: all 178 pairs cross a repository boundary. Not one is a within-repo copy. Whatever ran at L1 clearly caught duplicates inside a repository and did not catch the same generated or vendored file appearing in thousands of unrelated ones.

To be fair to the pipeline: "the highest ranked 50% are retained" means a group of four near-duplicates is supposed to keep two. Retaining exact copies is the documented behaviour, not a failure of it. The honest framing is that "near-duplicate deduplication" in this pipeline halves duplicate groups rather than collapsing them, and the card does say so if you read the sentence carefully. What the card does not say is that the residue is 3.2% of rows in a 5,500-row sample, or that it is concentrated in framework boilerplate: Flutter's generated windows/runner/utils.cpp five times, Laravel's stock config/database.php seven times.

There is one more thing in that data I did not go looking for. Of the 106 duplicate groups, 67 carry byte-identical content with different scores attached:

windows/runner/utils.cpp, 1,788 bytes, 5 identical copies

row          3   ambedgar777/quiz_flutter          q 6.763549327850342   algo 0.9988085031509399
row  4,658,133   Emad-Pro/Mini-Wall-Social-App     q 6.760730266571045   algo 0.9987998008728027
row 13,974,204   samir802/fileformatting           q 6.760730266571045   algo 0.9987998008728027
row 13,974,239   jackson338/gemini_goals           q 6.760730266571045   algo 0.9987998008728027
row 18,632,256   dev-ravan/hive_database_flutter   q 6.760730266571045   algo 0.9987998008728027

Same bytes in, different score out. The card says the framework "reuses precomputed semantic embeddings across role, relevance, and quality models", which makes this a statement about the embedding pass: identical inputs did not produce identical embeddings. The usual cause is batch-shape-dependent reduction order on a GPU, and the magnitude here — 3e-3 on a 0–10 scale, 9e-6 on a 0–1 scale — is exactly float noise, not a different model. It does not move any file across a threshold. It does mean the selection is not reproducible: re-run the pass and a file sitting on the 3.0 quality boundary could land on either side.

No group disagreed about the category label, so the role model at least is stable at the granularity that matters.

L3 is not one exercise per ALGO file

The overview figure labels L3's first stage "ALGO Seed" and the card says the protocol "converts each algorithmically relevant implementation selected by L2 into a programming exercise". That reads as a map from ALGO files to exercises, and it isn't one.

R is the split where both sides of the comparison are exact. /statistics reads all 843,971 rows of L2/r ("partial": false) and reports 162,047 ALGO files, 19.2005%. /size reports 304,016 rows in L3/r. That is 1.876 exercises per ALGO file, from two exact counts with nothing extrapolated.

Every language runs the same way, from 1.05× in Java to 2.08× in PHP. And my samples show why: across the 500 sampled L3/php rows, only 347 distinct raw_content values appear, while all 500 full_content values are distinct. The same source file was turned into several different exercises. That is a sensible thing to do — and "each implementation" is the wrong description of it.

The benchmark numbers, all of them

Since I have been unkind about the prose, the arithmetic deserves the same treatment. UltraData-Code has no tech report — the card's own header link reads "Tech Report (Coming Soon)" — so every benchmark number in this release exists only as prose next to a PNG of a table. That is a bad place for numbers to live, and it makes them very easy to check.

expand

expand

Thirteen numbers quoted in the card's prose, thirteen subtractions against its own tables:

Thirteen for thirteen. Nothing rounded up, nothing quoted from a different table. That is worth stating plainly, because it is the part of this release that is done properly and the part nobody will check.

Two footnotes on the same tables. First, the L2 block of the multilingual table contains Ultra-Stack-v3, which beats UltraData-Code-L2 on MultiPL-E by 2.37 points (22.75 vs 20.38) — the table even bolds it. The card's L2 bullet compares against Stack-Edu and does not mention it. Reporting the win over the older external baseline and skipping the loss to the in-house one is a choice, and it is the one place the card selects its comparison.

Second, the table's bottom row is labelled UltraData-Code-L3 while the prose calls the same number "the equal-token L2 + L3 mixture". The figure's legend says L2+L3. Two of the three agree and the odd one out is the row label, so the prose is right and the table is mislabelled — but you need the figure to know that.

The licence says two things

The card YAML declares license: apache-2.0. The repo ships an 11,335-byte LICENSE that is the verbatim Apache License 2.0, whose §2 grants "a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce … and distribute the Work".

Three paragraphs after citing that licence, the README says:

No unauthorized unchanged redistribution: Without prior written permission from the original authors (or this organization), any institution, organization, or third-party platform is strictly prohibited from directly reposting, mirroring, re-hosting, or commercially repackaging and republishing any artifacts of this project in any form.

The Chinese card carries the same sentence, so it is not a translation slip. Whatever that paragraph is, it is not Apache-2.0, and the Hub badge, every mirror, and every automated licence scan will read the YAML and see a permissive corpus.

The provenance section is more careful, and more honest than most: "Public availability of a repository is not a grant of redistribution, commercial use, or training rights", and "Users must also comply with the LICENSE of each source repository." That is the right thing to say. It is also impossible to act on for half the release. Compare the two schemas:

L2 row:  uuid, repo_name, relative_path, content, category,
         algo_rel_score, quality_score

L3 row:  uuid, content, content_format, raw_content, task, analysis,
         solution, test, full_content, full_content_format

L2 carries repo_name and relative_path, so "go read the source repository's licence" is a thing you can do — the duplicate ledger above names a repository for every row it lists. L3 carries raw_content, which is the verbatim source file, and drops both. That is 79.0B tokens of third-party source code redistributed with the attribution removed, under a card that instructs you to comply with each source file's licence. The uuid column exists in both and might join them; I could not test it, because the datasets-server /filter endpoint returns HTTP 500 for this dataset even though /is-valid advertises "filter": true.

One more gap. L3 is roughly 137B tokens of model-generated text. The card's Acknowledgements name exactly one model — Qwen3-Embedding-0.6B, used for L2's file embeddings — and never name the model that generated L3. Without that, nobody can check whether the generating model's terms of use permit redistributing its output, which is the single most common way a synthetic corpus becomes unusable.

What I could not check

Three things, stated so nobody mistakes them for measured:

• L0's "approximately 192 million public GitHub repositories." Nothing is released at L0 or L1. There is no artifact, so there is no check. The number is not implausible for a full public-GitHub crawl; it is simply unverifiable from this release.
• The exact corpus token count. My token numbers are extrapolations with a validated error term (0.07% on the byte cross-check), not exact counts, and they depend on the MiniCPM5-2B tokenizer. A different tokenizer moves every number in the same direction by roughly the same factor. The row counts and byte totals are exact.
• The true duplicate rate. 3.24% is what a 5,500-row sample of 266,878,376 rows caught. It is a floor. The real rate could be higher; it cannot be lower.

The arXiv link the card uses for its framework, 2602.09003, I did verify independently rather than trusting the card: it resolves to "Data Science and Technology Towards AGI Part I: Tiered Data Management", submitted 9 February 2026, seventeen authors including Xu Han, Zhiyuan Liu and Maosong Sun. It is the L0–L4 framework paper, not a paper about this corpus. (The sibling openbmb/UltraData-Math card still carries arxiv: xxxx.xxxxx in its YAML, which is the placeholder somebody forgot.)

The verdict

This is a serious release. 348 million rows, 1.2 TB, eleven languages, both selection models' outputs shipped inside the data so the selection can be audited by anyone with a curl and an afternoon — which is more transparency than most corpora of this size offer, and the reason this article was possible at all. Every benchmark delta it quotes is exactly right.

The failures are all of description. "~150B tokens" is one column out of ten and the card never says which. "Algorithmically relevant" is 26.6% ALGO. "Each algorithmically relevant implementation" is 1.88 exercises per implementation in the one split where both numbers are exact. "Near-duplicate deduplication" leaves Laravel's default config in seven times. And a corpus cannot be Apache-2.0 and also prohibited from being mirrored.

None of that makes the data worse. It makes the card worse than the data, which is the easier of the two problems to fix.