MiniMax Sparse Attention: let each query pick its own blocks

2026-07-04 · 6 min · attention · inference · mixture-of-experts

Why read this

How MSA's index branch picks 16 blocks per GQA group, why a fixed 2,048-token budget pays off only at long context, and where the converted model regresses.

A lasting reference
A new technique
Open code or weights

LLM architectureNeeds datacenter GPUsCustom licenceResearch paper

How this was scored

Is it new?: 2 of 3: A real new idea, method or capability
Can I trust it?: 1 of 3: Spot-checks a few numbers
Can I run it?: 2 of 3: Open code or weights with real limits
Will I understand it?: 2 of 3: Mechanism from first principles with figures
Can I act on it?: 1 of 3: General advice
Will it last?: 2 of 3: A reference for a year or more
Does it affect many?: 1 of 3: A specialist community
Only here?: 1 of 3: Some original analysis

Score 52 of 100, ranked 455 of 577 rated articles. Each question is answered 0–3 by hand, and a 3 is rare. How articles are scored

The thing that makes long context expensive is that, under full attention, every query token reads every past token. The KV cache that stores those keys and values grows linearly with sequence length, and the attention itself scales with it — so a 1M-token context is brutal to serve. The field's answer has been to make attention cheaper: linear-attention hybrids like HydraHead, sliding windows like MiMo-V2-Flash, or KV compression like TurboQuant. MiniMax Sparse Attention (MSA) takes a different route: keep attention exact, but let each query attend to only a small, learned subset of the context.

The trick is to think in blocks. Partition the past KV into fixed blocks of 128 tokens; for each query, a cheap index branch scores every block, keeps only the highest-scoring few, and the main attention runs — exactly — over just those. Scrub the query and watch which blocks it actually reads:

sparse attention · one query, its blocksillustrative

group

stage

attends 5 of 18 blocks · 640 tokens

query position (drag)

However far back the context runs, the query attends to a fixed handful of blocks — top-4 by index score, plus its local block. Flip the GQA group and the arrows swing to different blocks over identical keys and values: each group of query heads keeps its own sparse view of the past. (Real config: block 128, top-16 = 2,048 tokens per query.)

Two design choices make this work. First, the selection happens per GQA group: MiniMax uses grouped- query attention with 64 query heads tied to 4 KV heads, so there are 4 groups, and each group picks its own blocks off the same keys and values — flip the group above and the arrows swing. Second, the index is trained, not heuristic: an auxiliary KL loss aligns the index branch's block scores with the main attention's true distribution over blocks, with a stop-gradient so it only trains the tiny index projections and never disturbs the backbone.

The mechanism, precisely

Each attention layer splits into two branches:

Index branch (selection). It adds one lightweight index-query head per group and a single shared index-key head. For query i it computes Q_idx · K_idxᵀ, pools those token scores to the block level by max-pooling, and takes the top-k blocks — deployed as block size 128, k = 16, so a fixed budget of 2,048 tokens per query. The block the query sits in (the local block) is always kept.
Main branch (compute). Given the selected block indices, it runs exact attention over only those blocks. Because every head in a group reuses the same block set, KV reads stay block-contiguous — which is what lets a custom kernel turn the FLOP savings into real wall-clock speedup.

MSA architecture diagram. Left, an Index Branch takes hidden states through linear projection, norm and RoPE to index query and key heads, computes a score matrix, applies block max pooling, and emits a Top-K block index. Right, a Main Branch selects those Top-K KV blocks and runs exact sparse attention over Q, K, V. Far right, two attention-mask grids for Query Group 1 and Query Group 2 show different selected key blocks per group. — A lightweight Index Branch scores KV blocks (Q·Kᵀ → block max pool → top-k); the Main Branch runs exact attention over only the selected blocks. Each GQA group selects its own blocks, so Group 1 and Group 2 attend to different long-range keys (paper, Figure 1).

Why the fixed budget is the whole point

Because every query attends to the same 2,048 tokens no matter how long the context is, the fraction of the context each query reads collapses as context grows — but, honestly, the measured speedup is much smaller than that fraction would suggest, because the index branch still scans every block. Slide the context length and watch both numbers:

fixed budget · a smaller slice of a bigger contextmeasured on H800 (paper, Fig. 4)

context length (drag) · 1M

The fraction attended falls to 0.2% at 1M — but MSA is not 500× faster. The measured win is 14.2× prefill / 7.6× decode at 1M, because the Index Branch still scores every block (a small linear cost) and block-sparse memory access is less regular than dense. The gap between the theoretical 28.4× FLOP reduction and the wall-clock number is exactly that overhead — and the whole advantage is a long-context phenomenon: at 32k it's barely 1.6×.

Three line charts comparing GQA (blue) and MSA (green) as sequence length grows from 32k to 1M. Left: per-token attention FLOPs — GQA rises steeply while MSA stays nearly flat, annotated 28.4x reduction at 1M. Middle: prefilling latency, annotated 14.2x speedup. Right: decoding latency per token, annotated 7.6x speedup. — Efficiency vs GQA at matched head config (64 query / 4 KV heads; MSA block 128, k=16, 2,048-token budget). At 1M tokens on H800: 28.4× per-token attention-FLOP reduction, 14.2× prefill and 7.6× decode wall-clock speedup (paper, Figure 4).

The numbers

MSA is trained two ways — from scratch (MSA-PT) and by converting a full-attention checkpoint (MSA-CPT) — and compared against MiniMax's own GQA full-attention model at a matched 3T-token budget. The headline is parity, not a free lunch: MSA holds or slightly beats full attention on most of a 28-benchmark suite, with real regressions on a few. The efficiency, meanwhile, is decisive at long context:

Against MiniMax's own full-attention model (values in parentheses), MSA-PT holds or edges ahead — RULER-8K 84.2 (79.8), GSM8K 77.7 (76.2), MMLU 67.2 (67.0), HumanEval 64.0 (61.0), VisualWebBench 68.4 (55.6):

MSA-PT vs full attention (%)

RULER-8K

84.2%

GSM8K

77.7%

MMLU

67.2%

HumanEval

64%

VisualWebBench

68.4%

050100

The efficiency wins that motivate all of this, at 1M context on H800:

Speedup vs GQA full attention at 1M context (×)

Attn FLOPs (theory)

28.4×

Prefill

14.2×

Decode

7.6×

0102030

A few honesty notes. The baseline is provider-selected and internal — MSA vs MiniMax's own GQA model, not against other sparse-attention methods (NSA, MoBA) or external frontier models. Quality is "on par," and the converted MSA-CPT variant does trail full attention on some tasks (GSM8K 73.7 vs 76.2, HumanEval 57.9 vs 61.0, HELMET-128K −0.60) — the fixed budget shows up as small losses on retrieval-heavy long-context tasks. And the striking efficiency figures are the 1M extreme with a fixed 2,048-token budget on a specific head config; at 32k the advantage is barely 1.6×. The 28.4× is a theoretical FLOP count read off a chart, not measured throughput — the honest wall-clock numbers are 14.2× and 7.6×.

The take

MSA's contribution is that it makes selection a first-class, trainable part of attention rather than a bolt-on. The block granularity is the quiet key: picking whole 128-token blocks (not individual tokens) keeps memory access regular enough that a kernel can actually realize the savings, and sharing the selection across a GQA group keeps it cheap. Set against the other long-context playbooks — linear attention trades exactness for O(1) state; sliding windows drop the far past; KV quantization shrinks each entry — MSA keeps attention exact and full-range and simply reads less of it, chosen per query and per group. Whether a fixed 2,048-token budget holds up as tasks demand genuinely global reasoning is the open question the retrieval regressions hint at; but as a way to serve a 1M context at a fraction of the cost while staying on the full-attention quality curve, it's a clean, well-engineered bet. The production model, MiniMax-M3, ships with it (open weights, minimax-community license).

Built on MiniMax Sparse Attention (Lai, Xu, Yang et al.; MiniMax, 2026) and the MiniMax-M3 release. Benchmark and efficiency figures are quoted from the paper (the 109B-total / 6B-active experimental model; block size 128, top-16); the interactive diagrams are illustrations of the mechanism. Speedups are measured on H800 at 1M context.

The mechanism, precisely

Why the fixed budget is the whole point

The numbers

The take

Cite this article