2026-07-22 · 4 min · systems · tokenization · performance · open-source · explainer
Tokenization is the step nobody thinks about. Before a language model sees a single token, terabytes
of raw text have to be turned into integer IDs — and if you've ever run that job over a real corpus,
you know it's slower than it has any right to be. Gigatoken
(Marcel Rød) is a drop-in tokenizer that does it at gigabytes per second — up to roughly 1000×
faster than HuggingFace's tokenizers.
The detail that makes this interesting: it isn't a Python-vs-Rust story. HuggingFace tokenizers and
OpenAI's tiktoken are already multithreaded Rust. Gigatoken beats them by a factor of a thousand
anyway — which means the win is algorithmic, not a language swap.
Why a regex was the bottleneck
A BPE tokenizer does two jobs. First it pretokenizes — splits the text into word-ish chunks, almost universally by running a big regular expression over every byte. Then it applies the BPE merges within each chunk. Everyone pictures the merges as the expensive part; in practice the regex pretokenization is the majority of the wall-clock. Toggle it:
The trick isn’t more threads — HF tokenizers and tiktoken already run multithreaded Rust. It’s killing the regex that dominates the pass and turning repeated words into lookups.
Gigatoken's two moves both target that. It replaces the regex with a hand-written SIMD scanner that performs the exact same split at over 2 GB/s per thread, and it caches pretoken→token mappings, so any word it has already encoded becomes a lookup instead of a re-run of BPE. Real text is mostly repeated words, so the cache hits constantly. Add minimal Python round-trips and threads that barely touch each other, and you get the numbers below.
The numbers, per CPU
This is throughput encoding an 11.9 GB slice of OpenWebText, with the speedup over HuggingFace beside each bar. Note the honest split baked into the colors:
BPE tokenizers — GPT-2, Llama, Qwen, DeepSeek, GPT-OSS, and friends — hit ~20+ GB/s on a big EPYC and clear three-digit speedups. SentencePiece-based ones (Gemma, Mistral, CodeLlama) are the weak spot the author flags openly: still faster, but ~10–20×, because Gigatoken hasn't optimized that path. And the hardware matters as much as the tokenizer: a 144-core EPYC does GPT-2 at 24.5 GB/s, an M4 Max laptop at 8.8 GB/s (its best speedups actually exceed 1000× because HF is slower there too), and a single 8-core desktop Ryzen still lands around 100×.
What "gigabytes per second" buys you
Numbers this large stop meaning anything without a yardstick, so here's one: at the EPYC's rate you could tokenize all of Common Crawl — about 130 trillion tokens, effectively the whole public internet — in just under 6.5 hours. The same job on HuggingFace's tokenizer runs for the better part of a year. Drag the dataset size:
That's the real point. Tokenization is pure overhead on the path to training — you pay it every time you change a vocab, re-shuffle a corpus, or add data — and a tokenizer that runs at disk speed turns a multi-day preprocessing job into a coffee break.
Using it
Two modes. Compatibility mode is the drop-in: wrap an existing tokenizer and it behaves like the
original, output matched exactly (gt.Tokenizer(hf_tokenizer).as_hf() or .as_tiktoken()) — a bit of
speed traded for bit-for-bit parity. The Gigatoken API is the fast path, letting the Rust side
read files directly and skip Python overhead entirely. You can benchmark any HuggingFace tokenizer
against your own data without installing anything:
uvx --with tokenizers gigatoken bench 'openai-community/gpt2' owt_train.txt \
--validate --doc-separator "<|endoftext|>"It's honest about the edges, too: SentencePiece is under-optimized, WordPiece isn't supported yet, Windows is untested (use WSL), and there's still ABI3 overhead the author expects to claw back another 2× from. The README even carries an AI-use disclosure noting most of the code was hand-written, with AI help mainly for the user-facing API and porting SIMD strategies across AVX-512/AVX2/NEON.
The take
Gigatoken is a clean reminder that "already optimized" is not the same as "optimal." A step everyone had mentally checked off as solved — it's Rust, it's threaded, move on — was still leaving a 1000× on the table, because the actual hot loop (a regex nobody questioned) had never been rewritten for the hardware. It won't change what your model learns. It will change whether the tokenizer is ever the thing you're waiting on again.
Source: the Gigatoken README and benchmarks (Marcel Rød, 2026). Throughput measured on OpenWebText across EPYC 9565, Apple M4 Max, and Ryzen 9800X3D CPUs; the figure is the project's, the interactives are mine.