Kimi K3: a 2.8T open model that turns compute into intelligence 2.5× better

Moonshot's Kimi K3 is the largest open model anyone has shipped: 2.8 trillion parameters, about 50B active per token, a 1-million-token context, multimodal, released 2026-07-16 as open-weight. On Moonshot's own suite it beats every other model it was tested against on coding and agentic work, and trails only the two strongest proprietary systems — Claude Fable 5 and GPT-5.6 Sol. It is positioned, not unfairly, as Opus-4.8-class capability at Sonnet-5-class pricing.

The interesting part is not the parameter count. It is how the parameters are spent. K3 is built on two attention changes — Kimi Delta Attention (KDA) and Attention Residuals (AttnRes) — that rework how information flows across sequence length and across depth, and it scales up MoE sparsity hard: it activates 16 of 896 experts per token, inside a Stable LatentMoE framework. Together with refined training and data recipes, those structural changes yield roughly a 2.5× improvement in overall scaling efficiency over K2 — the model converts compute into intelligence more effectively. This piece is a first-principles tour of each piece, why it is new, and what a model like this actually costs to build.

Read the block bottom-to-top: the hidden state passes through the attention sublayer (Gated MLA + KDA), Attention Residuals reach back to earlier depths, and Stable LatentMoE routes the token to 16 of 896 experts before the block emits its output. Four ideas, each doing a specific job. Take them one at a time.

Kimi Delta Attention: constant-size memory over a million tokens

Ordinary softmax attention keeps a KV cache that grows by one entry per token. At a 1M-token context that cache is the whole ballgame: decoding is memory-bound on a cache that scales with sequence length, and it only gets heavier as the context fills.

KDA is a gated delta-rule linear attention. Instead of a growing cache it keeps a fixed-size recurrent state $S_t$ that each token updates in place: it erases a little of the old state (a gated decay) and writes the new key/value association (the delta rule). A compact way to write the family is

where $g_t$ is the per-channel gated decay (the erase), $\beta_t\, k_t v_t^{\top}$ is the written delta, and $o_t$ reads the state with the query $q_t$. The state $S_t$ is a fixed $d \times d$ matrix — its size does not depend on how many tokens came before. Scrub the recurrence and watch the state stay constant-size while a softmax cache piles up:

That constant-size state is what makes a genuine 1M context tractable, and it is why Moonshot reports up to 6.3× faster decoding in million-token contexts. It is not free — a linear-attention state is a lossy summary, not a perfect record, so K3 interleaves KDA with full-attention layers (via Gated MLA) to keep exact recall where it matters. KDA also breaks the assumptions of conventional prefix caching, so Moonshot contributed a KDA implementation to the vLLM community to make serving practical.

Attention Residuals: selective retrieval across depth

The second change is about depth, not length. A plain residual stream forces every layer to add the same accumulated state from the layer just below it, so representations from far-earlier depths get smeared together as they climb the stack. Attention Residuals let a layer instead selectively retrieve representations from specific earlier depths — a learned read across depth rather than a uniform accumulation. Toggle the two modes and scrub the current layer:

The payoff Moonshot reports is concrete: about 25% higher training efficiency at under 2% additional cost. That ratio is the tell — a cheap structural change that improves gradient flow and lets the stack go deeper without the usual degradation, which is exactly the kind of lever that compounds into the headline 2.5× scaling number.

Alongside these, the attention sublayer uses Gated MLA — Multi-head Latent Attention with a learnable gate that controls activation and sharpens attention selectivity — and the MLP nonlinearity is a Sigmoid Tanh Unit (SiTU), chosen for stable activation dynamics at 2.8T scale. Small pieces, but at this size "stable" is load-bearing.

Stable LatentMoE: 16 of 896, and why that is hard

Here is the aggressive part. K3's feed-forward is a mixture of experts with 896 experts, of which only 16 fire for any given token — an activation ratio of 1.8%. That sparsity is what lets a 2.8T-parameter model activate only ~50B parameters per token, so the compute per token is that of a ~50B model while the knowledge capacity is that of a 2.8T one. The experts are latent — they operate in a learned latent space rather than through hand-tuned token-to-expert heuristics. Scrub a few tokens and watch the selected 16 change:

At 1.8% activation, two problems that are mild in a denser MoE become first-order. Routing: which 16 experts you pick per token has to be learned well, or the model wastes most of its capacity. Load balance: if a few experts hog the tokens, the rest never train, and the effective model collapses to something far smaller than 2.8T. Standard MoEs fight this with an auxiliary load-balancing loss and a sensitive balance coefficient — one more hyperparameter that, tuned wrong, either destabilizes training or lets experts collapse.

Quantile balancing: no auxiliary loss, no knob

K3's answer is Quantile Balancing: derive each expert's allocation directly from the quantiles of its router scores. Set a target quantile — keep the top $q$ fraction of tokens by score — and every expert serves the same fraction of tokens by construction, with no auxiliary loss and no balance coefficient to tune. Drag the quantile and flip to the aux-loss regime to see the imbalance it removes:

The systems half of the story matters just as much. K3 uses a fully balanced expert-parallel training method with static shapes and no host synchronization. That is not a throwaway detail: variable expert loads normally produce variable tensor shapes, which force recompilation and host-side synchronization that stalls a large cluster. Quantile balancing gives every expert the same load, so the shapes are static, so the expert-parallel pipeline runs without host sync — the difference between 16-of-896 routing being a nice idea and being trainable at 2.8T.

With all four pieces on the table, here is the paper's module-level picture redrawn: the Stable LatentMoE and KDA blocks in full detail on the left, and on the right the Block Attention Residuals backbone — where each module's output flows through an α gate that can read every earlier block and the embedding, not just the layer below it.

Turning compute into intelligence

Stack it up — KDA's cheap long-context memory, AttnRes's cheap depth, LatentMoE's extreme-but-stable sparsity, plus refined training and data recipes — and Moonshot's headline is a ~2.5× improvement in overall scaling efficiency over K2. Concretely: K3 reaches the same capability at roughly 1/2.5 the training compute. Drag the capability marker:

That is the number that actually matters. "2.8 trillion parameters" is a spec-sheet figure; "2.5× more capability per FLOP" is an engineering result. It is what lets an open lab, working around compute limits, ship a frontier-adjacent model without a frontier-sized compute bill.

What it would take to train it

So what does building a 2.8T-A50B model actually cost? The sparsity helps here too: training compute for an MoE scales with the active parameters, not the total, so K3's per-token training FLOPs are those of a ~50B model. The standard estimate is

with $N_{\text{active}} \approx 50\text{B}$ and $D$ the number of training tokens. Moonshot has not published K3's exact token budget; for reference, K2 was trained on 15.5T tokens with the Muon optimizer. Plug in a frontier-scale token budget and pick a cluster — the estimate is millions of GPU-hours and weeks of wall-clock:

Three things make that estimate achievable rather than merely large:

• Per-Head Muon. K3 extends the Muon optimizer to optimize each attention head independently — more adaptive updates at scale, on top of Muon's already better compute-efficiency than AdamW for this regime.
• MXFP4 / MXFP8 quantization-aware training. From the SFT stage onward, K3 trains with MXFP4 weights and MXFP8 activations. The model is trained to be low-precision-native, which is why the full 2.8T weights fit in roughly 1.4 TB and why it is servable at all without a quality cliff.
• Static-shape expert parallelism. As above — quantile balancing plus static shapes and no host synchronization is what keeps a few-thousand-accelerator cluster busy instead of stalling on dynamic routing.

The memory reality is worth stating plainly: 2.8T parameters even at 4-bit is ~1.4 TB just for weights, before optimizer states and activations, so training and serving both demand tensor-, pipeline-, and expert-parallelism across many accelerators. This is a systems achievement as much as a modeling one.

The benchmarks

On coding, K3 is a clear #2-or-#3 behind Fable 5 and GPT-5.6 Sol, and ahead of everything else open or closed that Moonshot tested — with a few outright wins.

expand

On FrontierSWE it sits second, close behind Fable 5 and well ahead of the rest:

On Terminal Bench 2.1 it is effectively tied for first, and on the long-horizon SWE Marathon and Program Bench it is first outright:

The agentic and visual picture is similar — competitive across the board, and #1 on browsing:

expand

The pattern is consistent: K3 wins where the task is long-horizon and tool-heavy (SWE Marathon, Program Bench, BrowseComp, Automation Bench, SpreadsheetBench 2), and comes second to Fable 5 or GPT-5.6 Sol on the single-shot, knowledge-dense ones (GDPval and AA-Briefcase Elo, DeepSWE). For an open model, being in that conversation at all is the story.

What it costs to serve

The sparsity that makes K3 cheap to train makes it cheap to run. API pricing is $0.30 / MTok** on cache-hit input, **$3.00 / MTok on cache-miss input, and $15.00 / MTok output — and Moonshot reports cache-hit rates above 90% in coding workloads, so the effective input price is closer to the cheap number than the expensive one. MXFP4 weights keep the footprint at ~1.4 TB, and Moonshot recommends supernode configurations of 64 or more accelerators, with a MiniTriton CUDA-core roofline shown on NVIDIA L20. The KDA state and static-shape routing are what make that serving profile hold up at 1M context.

The take

Strip away the size record and what is genuinely new in K3 is a coherent set of efficiency bets: KDA buys a real 1M context with constant-size memory; AttnRes buys depth almost for free; Stable LatentMoE with Quantile Balancing buys 2.8T of capacity at ~50B of active compute and makes that extreme sparsity trainable without an aux-loss knob or host-sync stalls; Per-Head Muon and MXFP4/MXFP8 QAT make the whole thing converge and fit. The sum is the number that matters — ~2.5× more capability per FLOP than K2 — delivered in the open at 2.8T. It does not top the frontier, and it does not pretend to. What it proves is that the gap between open and closed is now measured in scaling efficiency, not in whether an open lab can build at frontier scale at all.

Sources: the Kimi K3 tech blog (architecture, Stable LatentMoE, Quantile Balancing, KDA, AttnRes, benchmarks, pricing) and Moonshot's reported figures. Benchmark numbers are quoted from Moonshot's charts; the training-cost figures are a first-principles estimate from $C \approx 6\,N_{\text{active}}\,D$ with clearly labeled assumptions, using K2's 15.5T-token budget as a reference. Interactive diagrams illustrate the mechanisms; the routing, loop, and cost visuals are illustrative.