# MiMo-V2-Flash: a 128-token window and one global layer in six
> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/mimo-v2-flash
> date: 2026-07-03
> tags: llm, mixture-of-experts, attention, agents, explainer
Xiaomi's **MiMo-V2-Flash** is, on the numbers, the **strongest open-source model for software
engineering** — 73.4 on SWE-Bench Verified, and the top score in its comparison table on
SWE-Bench Multilingual. It's a **309B-parameter Mixture-of-Experts** that activates just **15B**
per token. But the part worth an article isn't the leaderboard; it's the **attention design** that
lets a 309B model serve a 256K context quickly. MiMo doesn't use full attention, and it doesn't use
[linear attention like HydraHead](/articles/hydrahead) either. It interleaves **sliding-window** and
**global** layers — with a window so small it's almost provocative.
## The 5:1 hybrid, and a 128-token window
The backbone is 48 layers arranged into **8 hybrid blocks**, each block being **5 sliding-window
attention (SWA) layers followed by 1 global attention (GA) layer** — a 5:1 ratio. Each SWA layer
attends only to the previous **128 tokens**. That window is tiny on purpose; the report frames it as
an *inductive bias*: "smaller windows force the model to focus on local context... mitigat[ing]
overfitting." Two things keep 128 from being crippling:
- **Stacking compounds reach.** Like a stack of small convolutions, five SWA layers see far more than
128 tokens — each layer's window sits on top of the last, so information propagates several windows
back before you ever hit a global layer.
- **One global layer per block.** Every sixth layer attends to the *entire* sequence, mixing whatever
the local layers couldn't reach directly. Full context is restored periodically, at a sixth of the
cost of making every layer global.
A few architecture details worth having right: attention is **grouped-query (GQA), not MLA** — SWA
layers use 64 query / 8 KV heads, GA layers 64 / 4 — with **partial RoPE** on the first 64 dims and a
**learnable attention-sink bias** that the report credits with much of the hybrid's stability. The
MoE is **256 experts, top-8, no shared expert**; only the first block runs a dense FFN.
## Why the window is the point
The reason to accept a 128-token window is the **KV cache**. In a full-attention model, every layer's
cache grows with the context length. In MiMo, the five SWA layers per block stop growing once the
context passes 128 tokens — only the 1-in-6 global layers keep scaling. So at long context the cache
is dominated by that one-sixth:
The report claims "nearly a **6× reduction** in KV-cache storage and attention computation for long
contexts," and the interactive shows exactly where it comes from — the SWA layers flatten, the global
layers carry the linear term. This is the same economics as the
[KV-cache story in inference](/articles/how-llm-inference-works): the cache is what caps concurrency
and context, so shrinking it 6× is what makes a 309B model cheap to serve at 256K. And crucially,
long-context *quality* holds — MiMo posts the best open-source LongBench V2 score and near-perfect
needle retrieval (96.7% at 256K), evidence the hybrid isn't paying for its speed with reach.
## Multi-token prediction, for speed
MiMo also ships with **multi-token prediction** built in. It trains one MTP head during pretraining,
then replicates it into a **3-layer MTP module** for inference — a built-in draft model for
**self-speculative decoding**. The reported acceptance length reaches ~3.6 tokens and the measured
speedup is up to **2.6× decoding** (2.70× at batch size 96). If you want the mechanism, the
[multi-token prediction write-up](/articles/multi-token-prediction) covers exactly this
predict-several-verify-in-one-pass lineage; MiMo is a clean production instance of it.
## How it was trained
Pretraining is **27T tokens** in FP8 with the MTP objective, native 32K context, over three stages
(general → code-heavy with synthetic reasoning → context extension to 256K via RoPE base rescaling,
not YaRN). The post-training is the notable bit: it uses the **MOPD recipe** — multi-teacher on-policy
distillation, the same [MOPD from the recent arXiv digest](/arxiv/2026-06-30) that
[Agents-A1](/articles/agents-a1) also builds on — in three stages: SFT, domain-specialized RL, then
distilling several domain teachers into the student on its own rollouts. It's another data point that
on-policy multi-teacher distillation is becoming the default way to fuse agentic capabilities into one
deployable model.
## The numbers
The headline is agentic coding. On **SWE-Bench Multilingual** — resolving real GitHub issues across
languages — MiMo tops its entire comparison table, closed models included:
On **LiveCodeBench-v6** it's the best open model and edges GPT-5-High, trailing only Gemini-3.0-Pro:
And the one that validates the whole attention bet — **LongBench V2**, where the sliding-window hybrid
could have hurt but instead lands best-open, a whisker behind Claude:
It's not uniformly frontier — MiMo trails the reasoning leaders on HMMT (84.4), long-context MRCR
(45.7), and Humanity's Last Exam (22.1), and closed models still edge single-language SWE-Bench
Verified. The fair summary is: **the best open model for agentic software engineering and
long-context**, competitive with closed frontier systems on those axes, from a design that's cheaper
to serve. (The "150 tokens/second" and per-token pricing you'll see quoted are from Xiaomi's product
page, not the peer-reported tech report — treat them as marketing, not measurements.)
## The take
MiMo-V2-Flash is a satisfying piece of systems thinking more than a scaling flex. The bet is that most
of what attention does is *local* — so make most layers local and cheap (a 128-token window is an
aggressive way to commit to that), and buy back global reach with one full-attention layer per block
and a periodic reset. Stack MTP on top for decode speed and MOPD for capability, and you get a 309B
model that serves 256K context at a fraction of a full-attention model's KV cost while topping the
open-source agentic-coding board.
The through-line with [HydraHead](/articles/hydrahead) is worth noticing: both argue you shouldn't pay
full-attention cost on every layer, they just cut the budget differently — HydraHead per *head* by
interpretability, MiMo per *layer* on a fixed 5:1 schedule. The schedule is blunter, but it's simple,
it's proven at 309B, and the KV-cache math is undeniable. Open weights (Apache-2.0), MTP included — a
strong, honest release.
---
*Built on the [MiMo-V2-Flash Technical Report](https://arxiv.org/abs/2601.02780) (Xiaomi LLM-Core,
2025) and the [model release](https://github.com/XiaomiMiMo/MiMo-V2-Flash) (Apache-2.0, open weights +
3-layer MTP). Benchmark figures are quoted from the report's tables; throughput and pricing figures
are from Xiaomi's product page.*