2026-07-04 · 6 min · llm · attention · inference-optimization · mixture-of-experts · explainer
The thing that makes long context expensive is that, under full attention, every query token reads every past token. The KV cache that stores those keys and values grows linearly with sequence length, and the attention itself scales with it — so a 1M-token context is brutal to serve. The field's answer has been to make attention cheaper: linear-attention hybrids like HydraHead, sliding windows like MiMo-V2-Flash, or KV compression like TurboQuant. MiniMax Sparse Attention (MSA) takes a different route: keep attention exact, but let each query attend to only a small, learned subset of the context.
The trick is to think in blocks. Partition the past KV into fixed blocks of 128 tokens; for each query, a cheap index branch scores every block, keeps only the highest-scoring few, and the main attention runs — exactly — over just those. Scrub the query and watch which blocks it actually reads:
However far back the context runs, the query attends to a fixed handful of blocks — top-4 by index score, plus its local block. Flip the GQA group and the arrows swing to different blocks over identical keys and values: each group of query heads keeps its own sparse view of the past. (Real config: block 128, top-16 = 2,048 tokens per query.)
Two design choices make this work. First, the selection happens per GQA group: MiniMax uses grouped- query attention with 64 query heads tied to 4 KV heads, so there are 4 groups, and each group picks its own blocks off the same keys and values — flip the group above and the arrows swing. Second, the index is trained, not heuristic: an auxiliary KL loss aligns the index branch's block scores with the main attention's true distribution over blocks, with a stop-gradient so it only trains the tiny index projections and never disturbs the backbone.
The mechanism, precisely
Each attention layer splits into two branches:
- Index branch (selection). It adds one lightweight index-query head per group and a single shared
index-key head. For query
iit computesQ_idx · K_idxᵀ, pools those token scores to the block level by max-pooling, and takes the top-k blocks — deployed as block size 128, k = 16, so a fixed budget of 2,048 tokens per query. The block the query sits in (the local block) is always kept. - Main branch (compute). Given the selected block indices, it runs exact attention over only those blocks. Because every head in a group reuses the same block set, KV reads stay block-contiguous — which is what lets a custom kernel turn the FLOP savings into real wall-clock speedup.
Why the fixed budget is the whole point
Because every query attends to the same 2,048 tokens no matter how long the context is, the fraction of the context each query reads collapses as context grows — but, honestly, the measured speedup is much smaller than that fraction would suggest, because the index branch still scans every block. Slide the context length and watch both numbers:
The fraction attended falls to 0.2% at 1M — but MSA is not 500× faster. The measured win is 14.2× prefill / 7.6× decode at 1M, because the Index Branch still scores every block (a small linear cost) and block-sparse memory access is less regular than dense. The gap between the theoretical 28.4× FLOP reduction and the wall-clock number is exactly that overhead — and the whole advantage is a long-contextphenomenon: at 32k it's barely 1.6×.
The numbers
MSA is trained two ways — from scratch (MSA-PT) and by converting a full-attention checkpoint (MSA-CPT) — and compared against MiniMax's own GQA full-attention model at a matched 3T-token budget. The headline is parity, not a free lunch: MSA holds or slightly beats full attention on most of a 28-benchmark suite, with real regressions on a few. The efficiency, meanwhile, is decisive at long context:
Against MiniMax's own full-attention model (values in parentheses), MSA-PT holds or edges ahead — RULER-8K 84.2 (79.8), GSM8K 77.7 (76.2), MMLU 67.2 (67.0), HumanEval 64.0 (61.0), VisualWebBench 68.4 (55.6):
The efficiency wins that motivate all of this, at 1M context on H800:
A few honesty notes. The baseline is provider-selected and internal — MSA vs MiniMax's own GQA model, not against other sparse-attention methods (NSA, MoBA) or external frontier models. Quality is "on par," and the converted MSA-CPT variant does trail full attention on some tasks (GSM8K 73.7 vs 76.2, HumanEval 57.9 vs 61.0, HELMET-128K −0.60) — the fixed budget shows up as small losses on retrieval-heavy long-context tasks. And the striking efficiency figures are the 1M extreme with a fixed 2,048-token budget on a specific head config; at 32k the advantage is barely 1.6×. The 28.4× is a theoretical FLOP count read off a chart, not measured throughput — the honest wall-clock numbers are 14.2× and 7.6×.
The take
MSA's contribution is that it makes selection a first-class, trainable part of attention rather than a bolt-on. The block granularity is the quiet key: picking whole 128-token blocks (not individual tokens) keeps memory access regular enough that a kernel can actually realize the savings, and sharing the selection across a GQA group keeps it cheap. Set against the other long-context playbooks — linear attention trades exactness for O(1) state; sliding windows drop the far past; KV quantization shrinks each entry — MSA keeps attention exact and full-range and simply reads less of it, chosen per query and per group. Whether a fixed 2,048-token budget holds up as tasks demand genuinely global reasoning is the open question the retrieval regressions hint at; but as a way to serve a 1M context at a fraction of the cost while staying on the full-attention quality curve, it's a clean, well-engineered bet. The production model, MiniMax-M3, ships with it (open weights, minimax-community license).
Built on MiniMax Sparse Attention (Lai, Xu, Yang et al.; MiniMax, 2026) and the MiniMax-M3 release. Benchmark and efficiency figures are quoted from the paper (the 109B-total / 6B-active experimental model; block size 128, top-16); the interactive diagrams are illustrations of the mechanism. Speedups are measured on H800 at 1M context.