TurboQuant: rotate first, then quantize the KV cache

2026-07-02 · 7 min · llm · inference-optimization · quantization · kv-cache · explainer

The KV cache runs the economics of LLM serving: it grows linearly with context length, per layer, and it's what decides how many requests fit on a GPU. The obvious lever is to quantize it — store the cached keys and values in 2–4 bits instead of 16. The catch is that key and value vectors have outlier coordinates: a few dimensions carry most of the magnitude, so a fixed set of quantization levels either clips the outliers or wastes precision on the empty middle. The standard fix — calibrate a per-channel quantizer on sample data — doesn't work online, where new tokens stream in and you have no calibration set.

TurboQuant (Zandieh, Daliri, Hadian, Mirrokni — Google Research, arXiv 2504.19874) solves this with one idea: rotate the vector before you quantize it.

rotate, then quantize · one fixed quantizer for every vector

16 coordinates · shaded = the fixed quantizer range (±3/√d) · red = clipped outlier

reconstruction cosine similarity

0.9652

coordinates outside the quantizer

0 / 16

quantizer bit-width3-bit (8 levels)

Rotated: the energy is spread evenly, nothing clips, and the reconstruction error drops — using a quantizer set from the known post-rotation distribution alone, with no calibration data. Notice the cosine rise and the 0 clipped coordinates versus the raw case.

Why rotation is the whole trick

Multiply a vector by a random orthogonal matrix and you don't change its length or its inner products with other (equally rotated) vectors — but you do spread its energy evenly across all coordinates. After the rotation, the coordinates of a unit vector provably follow a known Beta distribution, $f(x) \propto (1 - x^2)^{(d-3)/2}$ on $[-1, 1]$ , which for large $d$ tightens to a Gaussian $\mathcal{N}(0, 1/d)$ . Every coordinate of every rotated vector looks the same, and there are no outliers left to clip.

That's what makes the quantizer data-free. Because you know the post-rotation distribution analytically, you can solve for the optimal scalar quantizer (Lloyd–Max: the levels that minimize expected squared error against that Beta density) once, ahead of time, and it's optimal for every vector you'll ever see — no calibration pass, which is exactly what an online KV cache needs. The paper proves this is near-optimal: TurboQuant's distortion is within a small constant factor (about 2.7×) of the information-theoretic lower bound for any vector quantizer, at every bit-width and dimension.

Two log-scale plots of quantization distortion versus bit-width (1 to 5 bits). Left: inner-product error for TurboQuant-prod and TurboQuant-mse sits between a green lower bound and a red upper bound. Right: mean squared error for TurboQuant-mse tracks just above the lower bound and hugs the upper bound. — TurboQuant's inner-product error (left) and MSE (right) stay wedged between the theoretical lower and upper distortion bounds at every bit-width — the near-optimal, data-free result the whole method rests on (paper, Figure 3).

The bias nobody mentions

There's a subtlety the paper is unusually careful about. An MSE-optimal quantizer minimizes reconstruction error — but attention doesn't reconstruct vectors, it takes inner products (query · key) and softmaxes them. And an MSE-optimal quantizer is biased as an inner-product estimator: on average its scores are systematically off, which quietly warps the attention weights. TurboQuant fixes this with a two-stage scheme — quantize with the MSE quantizer, then store the sign of the residual under a random projection (a 1-bit Quantized Johnson– Lindenstrauss transform, from the same authors' earlier QJL work). That 1-bit correction is exactly what cancels the bias, giving an unbiased inner-product estimate:

TurboQuant · rotate → quantize → correct

stage

1 · rotate

What: Multiply the key/value vector by a random orthogonal matrix (a Hadamard/QR rotation).

Why: It spreads energy evenly across coordinates, so each coordinate follows the same known Beta distribution — no outliers, no per-vector surprises.

Unbiasedness is the part that lets quality hold at aggressive bit-widths: the paper reports absolute quality neutrality at 3.5 bits per channel, and only marginal degradation at 2.5.

A 3-by-2 grid of needle-in-a-haystack recall heatmaps (depth percent vs token limit from 4k to 104k) for Llama-3.1-8B. SnapKV (0.858), PyramidKV (0.895) and KIVI (0.981) show scattered non-green cells where recall drops; PolarQuant (0.995), Full-Precision (0.997) and TurboQuant (0.997) are almost entirely green. — Needle-in-a-haystack recall for Llama-3.1-8B under a 0.25 KV-cache budget: despite more than 4x compression, TurboQuant matches the full-precision baseline (0.997) while SnapKV, PyramidKV and KIVI visibly lose recall (paper, Figure 4).

What it costs, what it buys

An implementation (0xsero/turboquant) wires this into vLLM and makes the tradeoff concrete. It allocates bits asymmetrically — keys get 3 bits with the unbiased inner-product quantizer (attention scores are precision-sensitive), values get 2 or 4 bits with simpler group quantization (value aggregation is more forgiving). The reconstruction quality splits cleanly along that line:

Reconstruction cosine similarity vs full precision

keys · 3-bit (unbiased)

values · 4-bit

values · 2-bit

0.94

00.51

Keys reconstruct essentially perfectly; 4-bit values are near-lossless; 2-bit values are where the quality actually gives (0.94), which is why the implementation recommends 4-bit values for anything sensitive. Net, it compresses the full-attention KV cache about 4.4×, and that turns straight into context length:

KV cache size vs context length · fp16 vs TurboQuant

context length

128K tok

fp16 cache

32.0 GB

TurboQuant

7.3 GB

max @ 40GB

160K→706K

Both grow linearly, but TurboQuant's slope is ~4.4× shallower. In a fixed 40 GB slice, fp16 tops out near 160K tokens; the compressed cache reaches 706K — the same GPU, a much longer context. (In practice the repo reports ~1.45–2.0× usable, since prefill still allocates a full cache and only full-attention layers compress.)

On the reported runs, that's 30 GB of KV cache freed on a 4-GPU RTX 5090 box and a 2.0× jump in max context (457K → 914K tokens) for a dense model; a MoE model with linear-attention layers gets less (1.45×), because those layers keep a recurrent state that doesn't compress. Throughput barely moves (+5.7% prefill, +3.1% decode) — this is a memory win, not a speed one, and the honest read is that the value comes from fitting longer contexts and bigger batches, not faster tokens. The current build also still allocates a full cache during prefill and only frees it afterward, and its hybrid decode path dequantizes history to fp32 each step — real limitations the repo names outright.

The same algorithm, three places

What makes TurboQuant worth an article isn't just the KV-cache result — it's that "rotate, then quantize with a data-free optimal quantizer" is a general vector-quantization primitive, and it's showing up in very different systems:

KV cache (0xsero/turboquant, above): compress the attention cache in vLLM for longer contexts.
Model weights (turbo-tan/llama.cpp-tq3): a TQ3 quantization type that applies the same rotate-then-quantize idea to weights in llama.cpp (see below).
Vector search (turbovec): the same rotation + Lloyd–Max + bit-packing, in Rust with SIMD kernels, as a FAISS-competitive similarity index — a 10M-vector corpus in 4 GB instead of 31. (Its own write-up is here.)

One paper, one primitive — a quantizer whose optimality comes from reshaping the data into a known distribution first rather than learning a codebook from samples — and it drops into inference caches, weight files, and ANN indexes alike.

TQ3 in llama.cpp: the same idea, on weights

The llama.cpp-tq3 fork adds TQ3_1S / TQ3_4S — 3-bit weight quantization types that run the TurboQuant pipeline (a Walsh–Hadamard rotation, then Lloyd–Max scalar quantization per block) on model weights. Worth clearing up a name collision: llama.cpp already ships TQ1_0 and TQ2_0, but those are ternary formats unrelated to this paper — the "TQ" match is coincidental. TQ3 is genuinely TurboQuant-based, and at ~3.5 bits per weight it hits Q4-class quality about 10% smaller (on Qwen3.5-27B, TQ3_4S measures a hair better perplexity than Q3_K_S at ~12.9 GiB), which is what lets a 27B model run on a 16 GB GPU.

The speedup is the interesting engineering twist. Because TQ3 blocks are 3-bit-after-rotation, they map cleanly onto FP4 tensor cores on Blackwell-class GPUs — the fork fuses the rotation into an FP4 activation quantizer and runs the matmul in FP4. Turning that path on roughly doubles prompt-processing throughput: on an RTX 3090, Gemma-12B goes 737 → 1,819 tok/s (+147%) and SuperGemma-26B 983 → 2,005 (+104%); on a DGX Spark (GB10), a 27B MTP model goes 360 → 920 tok/s (+155%). That's a rotate-then-quantize weight format turning a hardware FP4 unit into free speed — the same primitive, paying off a third way. (The exact llama-quantize invocation isn't documented in the repo yet; the types ship as pre-quantized models on Hugging Face.)

The take

The elegant part of TurboQuant is that the hard problem (outliers, calibration) is dissolved rather than fought. Instead of detecting and special-casing outlier channels, you rotate them away; instead of calibrating on data, you compute the optimal quantizer against the distribution the rotation guarantees. The QJL residual is the tasteful finish — a one-bit patch that turns a good reconstruction quantizer into an unbiased inner-product quantizer, which is the thing attention actually needs.

It's not magic: the KV-cache implementation is a memory win, not a throughput one, 2-bit values visibly degrade, and MoE/linear-attention models compress less. But the underlying result — near-optimal, data-free vector quantization with a formal distortion bound — is the kind of solid primitive that ends up everywhere, which is exactly what's happening.

Built on TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate (Amir Zandieh, Majid Daliri, Majid Hadian, Vahab Mirrokni; 2025), building on the authors' earlier QJL work. Implementation details and benchmarks are from 0xsero/turboquant (GPLv3) and turbo-tan/llama.cpp-tq3.