# TurboQuant: rotate first, then quantize the KV cache

> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/turboquant-kv-cache
> date: 2026-07-02
> tags: llm, inference-optimization, quantization, kv-cache, explainer
The [KV cache runs the economics of LLM serving](/articles/how-llm-inference-works): it grows
linearly with context length, per layer, and it's what decides how many requests fit on a GPU.
The obvious lever is to **quantize** it — store the cached keys and values in 2–4 bits instead
of 16. The catch is that key and value vectors have **outlier coordinates**: a few dimensions
carry most of the magnitude, so a fixed set of quantization levels either clips the outliers or
wastes precision on the empty middle. The standard fix — calibrate a per-channel quantizer on
sample data — doesn't work *online*, where new tokens stream in and you have no calibration set.

**TurboQuant** (Zandieh, Daliri, Hadian, Mirrokni — Google Research, [arXiv 2504.19874](https://arxiv.org/abs/2504.19874))
solves this with one idea: **rotate the vector before you quantize it.**

<Rotation />

## Why rotation is the whole trick

Multiply a vector by a random orthogonal matrix and you don't change its length or its inner
products with other (equally rotated) vectors — but you *do* spread its energy evenly across all
coordinates. After the rotation, the coordinates of a unit vector provably follow a known
**Beta distribution**, $f(x) \propto (1 - x^2)^{(d-3)/2}$ on $[-1, 1]$, which for large $d$
tightens to a Gaussian $\mathcal{N}(0, 1/d)$. Every coordinate of every rotated vector looks the
same, and there are no outliers left to clip.

That's what makes the quantizer **data-free**. Because you know the post-rotation distribution
analytically, you can solve for the optimal scalar quantizer (Lloyd–Max: the levels that minimize
expected squared error against that Beta density) *once, ahead of time*, and it's optimal for
every vector you'll ever see — no calibration pass, which is exactly what an online KV cache
needs. The paper proves this is **near-optimal**: TurboQuant's distortion is within a small
constant factor (about **2.7×**) of the information-theoretic lower bound for *any* vector
quantizer, at every bit-width and dimension.

<Figure
  src="/articles/turboquant-kv-cache/fig1.png"
  alt="Two log-scale plots of quantization distortion versus bit-width (1 to 5 bits). Left: inner-product error for TurboQuant-prod and TurboQuant-mse sits between a green lower bound and a red upper bound. Right: mean squared error for TurboQuant-mse tracks just above the lower bound and hugs the upper bound."
  caption="TurboQuant's inner-product error (left) and MSE (right) stay wedged between the theoretical lower and upper distortion bounds at every bit-width — the near-optimal, data-free result the whole method rests on (paper, Figure 3)."
/>

## The bias nobody mentions

There's a subtlety the paper is unusually careful about. An MSE-optimal quantizer minimizes
reconstruction error — but attention doesn't reconstruct vectors, it takes **inner products**
(query · key) and softmaxes them. And an MSE-optimal quantizer is *biased* as an inner-product
estimator: on average its scores are systematically off, which quietly warps the attention
weights. TurboQuant fixes this with a two-stage scheme — quantize with the MSE quantizer, then
store the **sign of the residual** under a random projection (a 1-bit *Quantized Johnson–
Lindenstrauss* transform, from the same authors' earlier QJL work). That 1-bit correction is
exactly what cancels the bias, giving an **unbiased** inner-product estimate:

<Pipeline />

Unbiasedness is the part that lets quality hold at aggressive bit-widths: the paper reports
**absolute quality neutrality at 3.5 bits per channel**, and only marginal degradation at 2.5.

<Figure
  src="/articles/turboquant-kv-cache/fig2.png"
  alt="A 3-by-2 grid of needle-in-a-haystack recall heatmaps (depth percent vs token limit from 4k to 104k) for Llama-3.1-8B. SnapKV (0.858), PyramidKV (0.895) and KIVI (0.981) show scattered non-green cells where recall drops; PolarQuant (0.995), Full-Precision (0.997) and TurboQuant (0.997) are almost entirely green."
  caption="Needle-in-a-haystack recall for Llama-3.1-8B under a 0.25 KV-cache budget: despite more than 4x compression, TurboQuant matches the full-precision baseline (0.997) while SnapKV, PyramidKV and KIVI visibly lose recall (paper, Figure 4)."
/>

## What it costs, what it buys

An implementation ([`0xsero/turboquant`](https://github.com/0xsero/turboquant)) wires this into
vLLM and makes the tradeoff concrete. It allocates bits **asymmetrically** — keys get **3 bits**
with the unbiased inner-product quantizer (attention scores are precision-sensitive), values get
**2 or 4 bits** with simpler group quantization (value aggregation is more forgiving). The
reconstruction quality splits cleanly along that line:

<BenchBars
  title="Reconstruction cosine similarity vs full precision"
  unit=""
  max={1}
  bars={[
    { label: "keys · 3-bit (unbiased)", value: 1.0, highlight: true },
    { label: "values · 4-bit", value: 0.997, highlight: true },
    { label: "values · 2-bit", value: 0.94 },
  ]}
/>

Keys reconstruct essentially perfectly; 4-bit values are near-lossless; 2-bit values are where
the quality actually gives (0.94), which is why the implementation recommends 4-bit values for
anything sensitive. Net, it compresses the full-attention KV cache about **4.4×**, and that turns
straight into context length:

<KvMemory />

On the reported runs, that's **30 GB of KV cache freed** on a 4-GPU RTX 5090 box and a **2.0×**
jump in max context (457K → 914K tokens) for a dense model; a MoE model with linear-attention
layers gets less (**1.45×**), because those layers keep a recurrent state that doesn't compress.
Throughput barely moves (**+5.7%** prefill, **+3.1%** decode) — this is a *memory* win, not a
speed one, and the honest read is that the value comes from fitting longer contexts and bigger
batches, not faster tokens. The current build also still allocates a full cache during prefill
and only frees it afterward, and its hybrid decode path dequantizes history to fp32 each step —
real limitations the repo names outright.

## The same algorithm, three places

What makes TurboQuant worth an article isn't just the KV-cache result — it's that "rotate, then
quantize with a data-free optimal quantizer" is a **general** vector-quantization primitive, and
it's showing up in very different systems:

- **KV cache** (`0xsero/turboquant`, above): compress the attention cache in vLLM for longer
  contexts.
- **Model weights** (`turbo-tan/llama.cpp-tq3`): a `TQ3` quantization type that applies the same
  rotate-then-quantize idea to *weights* in llama.cpp (see below).
- **Vector search** ([`turbovec`](/articles/turbovec)): the same rotation + Lloyd–Max +
  bit-packing, in Rust with SIMD kernels, as a FAISS-competitive similarity index — a 10M-vector
  corpus in 4 GB instead of 31. (Its own write-up is [here](/articles/turbovec).)

One paper, one primitive — a quantizer whose optimality comes from *reshaping the data into a
known distribution first* rather than learning a codebook from samples — and it drops into
inference caches, weight files, and ANN indexes alike.

### TQ3 in llama.cpp: the same idea, on weights

The `llama.cpp-tq3` fork adds `TQ3_1S` / `TQ3_4S` — **3-bit weight** quantization types that run the
TurboQuant pipeline (a Walsh–Hadamard rotation, then Lloyd–Max scalar quantization per block) on
model weights. Worth clearing up a name collision: llama.cpp already ships `TQ1_0` and `TQ2_0`,
but those are *ternary* formats unrelated to this paper — the "TQ" match is coincidental. TQ3 is
genuinely TurboQuant-based, and at ~3.5 bits per weight it hits **Q4-class quality about 10%
smaller** (on Qwen3.5-27B, `TQ3_4S` measures a hair *better* perplexity than `Q3_K_S` at ~12.9 GiB),
which is what lets a 27B model run on a 16 GB GPU.

The speedup is the interesting engineering twist. Because TQ3 blocks are 3-bit-after-rotation, they
map cleanly onto **FP4 tensor cores** on Blackwell-class GPUs — the fork fuses the rotation into an
FP4 activation quantizer and runs the matmul in FP4. Turning that path on roughly **doubles
prompt-processing throughput**: on an RTX 3090, Gemma-12B goes 737 → **1,819 tok/s** (+147%) and
SuperGemma-26B 983 → **2,005** (+104%); on a DGX Spark (GB10), a 27B MTP model goes 360 → **920
tok/s** (+155%). That's a rotate-then-quantize weight format turning a hardware FP4 unit into free
speed — the same primitive, paying off a third way. (The exact `llama-quantize` invocation isn't
documented in the repo yet; the types ship as pre-quantized models on Hugging Face.)

## The take

The elegant part of TurboQuant is that the hard problem (outliers, calibration) is dissolved
rather than fought. Instead of detecting and special-casing outlier channels, you rotate them
away; instead of calibrating on data, you compute the optimal quantizer against the distribution
the rotation guarantees. The QJL residual is the tasteful finish — a one-bit patch that turns a
good reconstruction quantizer into an unbiased *inner-product* quantizer, which is the thing
attention actually needs.

It's not magic: the KV-cache implementation is a memory win, not a throughput one, 2-bit values
visibly degrade, and MoE/linear-attention models compress less. But the underlying result —
near-optimal, data-free vector quantization with a formal distortion bound — is the kind of solid
primitive that ends up everywhere, which is exactly what's happening.

---

*Built on [TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate](https://arxiv.org/abs/2504.19874)
(Amir Zandieh, Majid Daliri, Majid Hadian, Vahab Mirrokni; 2025), building on the authors' earlier
QJL work. Implementation details and benchmarks are from [`0xsero/turboquant`](https://github.com/0xsero/turboquant)
(GPLv3) and [`turbo-tan/llama.cpp-tq3`](https://github.com/turbo-tan/llama.cpp-tq3).*