# Gigatoken: tokenizing at gigabytes per second

> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/gigatoken
> date: 2026-07-22
> tags: systems, tokenization, performance, open-source, explainer
Tokenization is the step nobody thinks about. Before a language model sees a single token, terabytes
of raw text have to be turned into integer IDs — and if you've ever run that job over a real corpus,
you know it's slower than it has any right to be. [Gigatoken](https://github.com/marcelroed/gigatoken)
(Marcel Rød) is a drop-in tokenizer that does it at **gigabytes per second** — up to roughly **1000×
faster** than HuggingFace's `tokenizers`.

The detail that makes this interesting: it isn't a Python-vs-Rust story. HuggingFace `tokenizers` and
OpenAI's `tiktoken` are *already* multithreaded Rust. Gigatoken beats them by a factor of a thousand
anyway — which means the win is algorithmic, not a language swap.

<Figure
  src="/articles/gigatoken/fig1.png"
  alt="Bar chart of GPT-2 tokenizer throughput on a 12 GB file on an Apple M4 Max: gigatoken 8.27 GB/s, tiktoken 61.5 MB/s, HuggingFace tokenizers 6.2 MB/s — the gigatoken bar dwarfs the other two."
  caption="GPT-2 on a 12 GB file, M4 Max: 8.27 GB/s vs 6.2 MB/s for HuggingFace — and the output is validated to match exactly (gigatoken README)."
/>

## Why a regex was the bottleneck

A BPE tokenizer does two jobs. First it **pretokenizes** — splits the text into word-ish chunks,
almost universally by running a big regular expression over every byte. Then it applies the **BPE
merges** within each chunk. Everyone pictures the merges as the expensive part; in practice the
**regex pretokenization is the majority of the wall-clock**. Toggle it:

<PretokenPipeline />

Gigatoken's two moves both target that. It replaces the regex with a hand-written **SIMD** scanner
that performs the exact same split at over 2 GB/s per thread, and it **caches pretoken→token
mappings**, so any word it has already encoded becomes a lookup instead of a re-run of BPE. Real text
is mostly repeated words, so the cache hits constantly. Add minimal Python round-trips and threads
that barely touch each other, and you get the numbers below.

## The numbers, per CPU

This is throughput encoding an 11.9 GB slice of OpenWebText, with the speedup over HuggingFace beside
each bar. Note the honest split baked into the colors:

<ThroughputBars />

**BPE tokenizers** — GPT-2, Llama, Qwen, DeepSeek, GPT-OSS, and friends — hit ~20+ GB/s on a big EPYC
and clear three-digit speedups. **SentencePiece-based** ones (Gemma, Mistral, CodeLlama) are the
weak spot the author flags openly: still faster, but ~10–20×, because Gigatoken hasn't optimized that
path. And the hardware matters as much as the tokenizer: a 144-core EPYC does GPT-2 at 24.5 GB/s, an
M4 Max laptop at 8.8 GB/s (its best speedups actually *exceed* 1000× because HF is slower there too),
and a single 8-core desktop Ryzen still lands around 100×.

## What "gigabytes per second" buys you

Numbers this large stop meaning anything without a yardstick, so here's one: at the EPYC's rate you
could tokenize **all of Common Crawl — about 130 trillion tokens, effectively the whole public
internet — in just under 6.5 hours.** The same job on HuggingFace's tokenizer runs for the better part
of a year. Drag the dataset size:

<CommonCrawlClock />

That's the real point. Tokenization is pure overhead on the path to training — you pay it every time
you change a vocab, re-shuffle a corpus, or add data — and a tokenizer that runs at disk speed turns a
multi-day preprocessing job into a coffee break.

## Using it

Two modes. **Compatibility mode** is the drop-in: wrap an existing tokenizer and it behaves like the
original, output matched exactly (`gt.Tokenizer(hf_tokenizer).as_hf()` or `.as_tiktoken()`) — a bit of
speed traded for bit-for-bit parity. The **Gigatoken API** is the fast path, letting the Rust side
read files directly and skip Python overhead entirely. You can benchmark any HuggingFace tokenizer
against your own data without installing anything:

```bash
uvx --with tokenizers gigatoken bench 'openai-community/gpt2' owt_train.txt \
    --validate --doc-separator "<|endoftext|>"
```

It's honest about the edges, too: SentencePiece is under-optimized, WordPiece isn't supported yet,
Windows is untested (use WSL), and there's still ABI3 overhead the author expects to claw back another
2× from. The README even carries an AI-use disclosure noting most of the code was hand-written, with
AI help mainly for the user-facing API and porting SIMD strategies across AVX-512/AVX2/NEON.

## The take

Gigatoken is a clean reminder that "already optimized" is not the same as "optimal." A step everyone
had mentally checked off as solved — it's Rust, it's threaded, move on — was still leaving a **1000×**
on the table, because the actual hot loop (a regex nobody questioned) had never been rewritten for the
hardware. It won't change what your model learns. It will change whether the tokenizer is ever the
thing you're waiting on again.

---

*Source: the [Gigatoken README and benchmarks](https://github.com/marcelroed/gigatoken#benchmarks)
(Marcel Rød, 2026). Throughput measured on OpenWebText across EPYC 9565, Apple M4 Max, and Ryzen
9800X3D CPUs; the figure is the project's, the interactives are mine.*