2026-07-03 · 7 min · mixture-of-experts · attention · agents
Why read this
Solidtop 85%Shows why five 128-token sliding-window layers per global layer cut the KV cache about 6x at 256K, with MiMo-V2-Flash's reported scores.
- Widely used
- A lasting reference
- A new technique
LLM architectureNeeds datacenter GPUsApache-2.0Practitioner model
How this was scored
- Is it new?
- 2 of 3: A real new idea, method or capability
- Can I trust it?
- 0 of 3: Restates claims
- Can I run it?
- 2 of 3: Open code or weights with real limits
- Will I understand it?
- 2 of 3: Mechanism from first principles with figures
- Can I act on it?
- 1 of 3: General advice
- Will it last?
- 2 of 3: A reference for a year or more
- Does it affect many?
- 2 of 3: A widely used model, tool or lab release
- Only here?
- 1 of 3: Some original analysis
Score 50 of 100, ranked 467 of 577 rated articles. Each question is answered 0–3 by hand, and a 3 is rare. How articles are scored
Xiaomi's MiMo-V2-Flash is, on the numbers, the strongest open-source model for software engineering — 73.4 on SWE-Bench Verified, and the top score in its comparison table on SWE-Bench Multilingual. It's a 309B-parameter Mixture-of-Experts that activates just 15B per token. But the part worth an article isn't the leaderboard; it's the attention design that lets a 309B model serve a 256K context quickly. MiMo doesn't use full attention, and it doesn't use linear attention like HydraHead either. It interleaves sliding-window and global layers — with a window so small it's almost provocative.
The 5:1 hybrid, and a 128-token window
The backbone is 48 layers arranged into 8 hybrid blocks, each block being 5 sliding-window attention (SWA) layers followed by 1 global attention (GA) layer — a 5:1 ratio. Each SWA layer attends only to the previous 128 tokens. That window is tiny on purpose; the report frames it as an inductive bias: "smaller windows force the model to focus on local context... mitigat[ing] overfitting." Two things keep 128 from being crippling:
- Stacking compounds reach. Like a stack of small convolutions, five SWA layers see far more than 128 tokens — each layer's window sits on top of the last, so information propagates several windows back before you ever hit a global layer.
- One global layer per block. Every sixth layer attends to the entire sequence, mixing whatever the local layers couldn't reach directly. Full context is restored periodically, at a sixth of the cost of making every layer global.
A few architecture details worth having right: attention is grouped-query (GQA), not MLA — SWA layers use 64 query / 8 KV heads, GA layers 64 / 4 — with partial RoPE on the first 64 dims and a learnable attention-sink bias that the report credits with much of the hybrid's stability. The MoE is 256 experts, top-8, no shared expert; only the first block runs a dense FFN.
Why the window is the point
The reason to accept a 128-token window is the KV cache. In a full-attention model, every layer's cache grows with the context length. In MiMo, the five SWA layers per block stop growing once the context passes 128 tokens — only the 1-in-6 global layers keep scaling. So at long context the cache is dominated by that one-sixth:
The report claims "nearly a 6× reduction in KV-cache storage and attention computation for long contexts," and the interactive shows exactly where it comes from — the SWA layers flatten, the global layers carry the linear term. This is the same economics as the KV-cache story in inference: the cache is what caps concurrency and context, so shrinking it 6× is what makes a 309B model cheap to serve at 256K. And crucially, long-context quality holds — MiMo posts the best open-source LongBench V2 score and near-perfect needle retrieval (96.7% at 256K), evidence the hybrid isn't paying for its speed with reach.
Multi-token prediction, for speed
MiMo also ships with multi-token prediction built in. It trains one MTP head during pretraining, then replicates it into a 3-layer MTP module for inference — a built-in draft model for self-speculative decoding. The reported acceptance length reaches ~3.6 tokens and the measured speedup is up to 2.6× decoding (2.70× at batch size 96). If you want the mechanism, the multi-token prediction write-up covers exactly this predict-several-verify-in-one-pass lineage; MiMo is a clean production instance of it.
How it was trained
Pretraining is 27T tokens in FP8 with the MTP objective, native 32K context, over three stages (general → code-heavy with synthetic reasoning → context extension to 256K via RoPE base rescaling, not YaRN). The post-training is the notable bit: it uses the MOPD recipe — multi-teacher on-policy distillation, the same MOPD from the recent arXiv digest that Agents-A1 also builds on — in three stages: SFT, domain-specialized RL, then distilling several domain teachers into the student on its own rollouts. It's another data point that on-policy multi-teacher distillation is becoming the default way to fuse agentic capabilities into one deployable model.
The numbers
The headline is agentic coding. On SWE-Bench Multilingual — resolving real GitHub issues across languages — MiMo tops its entire comparison table, closed models included:
On LiveCodeBench-v6 it's the best open model and edges GPT-5-High, trailing only Gemini-3.0-Pro:
And the one that validates the whole attention bet — LongBench V2, where the sliding-window hybrid could have hurt but instead lands best-open, a whisker behind Claude:
It's not uniformly frontier — MiMo trails the reasoning leaders on HMMT (84.4), long-context MRCR (45.7), and Humanity's Last Exam (22.1), and closed models still edge single-language SWE-Bench Verified. The fair summary is: the best open model for agentic software engineering and long-context, competitive with closed frontier systems on those axes, from a design that's cheaper to serve. (The "150 tokens/second" and per-token pricing you'll see quoted are from Xiaomi's product page, not the peer-reported tech report — treat them as marketing, not measurements.)
The take
MiMo-V2-Flash is a satisfying piece of systems thinking more than a scaling flex. The bet is that most of what attention does is local — so make most layers local and cheap (a 128-token window is an aggressive way to commit to that), and buy back global reach with one full-attention layer per block and a periodic reset. Stack MTP on top for decode speed and MOPD for capability, and you get a 309B model that serves 256K context at a fraction of a full-attention model's KV cost while topping the open-source agentic-coding board.
The through-line with HydraHead is worth noticing: both argue you shouldn't pay full-attention cost on every layer, they just cut the budget differently — HydraHead per head by interpretability, MiMo per layer on a fixed 5:1 schedule. The schedule is blunter, but it's simple, it's proven at 309B, and the KV-cache math is undeniable. Open weights (Apache-2.0), MTP included — a strong, honest release.
Built on the MiMo-V2-Flash Technical Report (Xiaomi LLM-Core, 2025) and the model release (Apache-2.0, open weights + 3-layer MTP). Benchmark figures are quoted from the report's tables; throughput and pricing figures are from Xiaomi's product page.