# Kimi K3: a 2.8T open model that turns compute into intelligence 2.5× better > Satyajit Ghana — Head of Engineering @ Inkers Technology > canonical: https://ai.thesatyajit.com/articles/kimi-k3 > date: 2026-07-17 > tags: llm, mixture-of-experts, linear-attention, kimi, scaling, explainer Moonshot's [Kimi K3](https://www.kimi.com/blog/kimi-k3) is the largest open model anyone has shipped: **2.8 trillion** parameters, about **50B active** per token, a **1-million-token** context, multimodal, released 2026-07-16 as open-weight. On Moonshot's own suite it beats every other model it was tested against on coding and agentic work, and trails only the two strongest proprietary systems — Claude Fable 5 and GPT-5.6 Sol. It is positioned, not unfairly, as Opus-4.8-class capability at Sonnet-5-class pricing. The interesting part is not the parameter count. It is *how the parameters are spent*. K3 is built on two attention changes — **Kimi Delta Attention (KDA)** and **Attention Residuals (AttnRes)** — that rework how information flows across sequence length and across depth, and it scales up MoE sparsity hard: it activates **16 of 896 experts** per token, inside a **Stable LatentMoE** framework. Together with refined training and data recipes, those structural changes yield roughly a **2.5× improvement in overall scaling efficiency** over K2 — the model converts compute into intelligence more effectively. This piece is a first-principles tour of each piece, why it is new, and what a model like this actually costs to build. Read the block bottom-to-top: the hidden state passes through the attention sublayer (Gated MLA + KDA), Attention Residuals reach back to earlier depths, and Stable LatentMoE routes the token to 16 of 896 experts before the block emits its output. Four ideas, each doing a specific job. Take them one at a time. ## Kimi Delta Attention: constant-size memory over a million tokens Ordinary softmax attention keeps a **KV cache** that grows by one entry per token. At a 1M-token context that cache is the whole ballgame: decoding is [memory-bound on a cache that scales with sequence length](/articles/how-llm-inference-works), and it only gets heavier as the context fills. KDA is a **gated delta-rule linear attention**. Instead of a growing cache it keeps a **fixed-size recurrent state** $S_t$ that each token updates in place: it *erases* a little of the old state (a gated decay) and *writes* the new key/value association (the delta rule). A compact way to write the family is $$ S_t = g_t \odot S_{t-1} + \beta_t\, k_t v_t^{\top}, \qquad o_t = S_t\, q_t $$ where $g_t$ is the per-channel gated decay (the erase), $\beta_t\, k_t v_t^{\top}$ is the written delta, and $o_t$ reads the state with the query $q_t$. The state $S_t$ is a fixed $d \times d$ matrix — its size does not depend on how many tokens came before. Scrub the recurrence and watch the state stay constant-size while a softmax cache piles up: That constant-size state is what makes a genuine 1M context tractable, and it is why Moonshot reports **up to 6.3× faster decoding** in million-token contexts. It is not free — a linear-attention state is a lossy summary, not a perfect record, so K3 interleaves KDA with full-attention layers (via Gated MLA) to keep exact recall where it matters. KDA also breaks the assumptions of conventional prefix caching, so Moonshot contributed a KDA implementation to the vLLM community to make serving practical. ## Attention Residuals: selective retrieval across depth The second change is about *depth*, not length. A plain residual stream forces every layer to add the same accumulated state from the layer just below it, so representations from far-earlier depths get smeared together as they climb the stack. **Attention Residuals** let a layer instead *selectively retrieve* representations from specific earlier depths — a learned read across depth rather than a uniform accumulation. Toggle the two modes and scrub the current layer: The payoff Moonshot reports is concrete: about **25% higher training efficiency at under 2% additional cost**. That ratio is the tell — a cheap structural change that improves gradient flow and lets the stack go deeper without the usual degradation, which is exactly the kind of lever that compounds into the headline 2.5× scaling number. Alongside these, the attention sublayer uses **Gated MLA** — Multi-head Latent Attention with a learnable gate that controls activation and sharpens attention selectivity — and the MLP nonlinearity is a **Sigmoid Tanh Unit (SiTU)**, chosen for stable activation dynamics at 2.8T scale. Small pieces, but at this size "stable" is load-bearing. ## Stable LatentMoE: 16 of 896, and why that is hard Here is the aggressive part. K3's feed-forward is a mixture of experts with **896 experts**, of which only **16** fire for any given token — an activation ratio of **1.8%**. That sparsity is what lets a 2.8T-parameter model activate only ~50B parameters per token, so the compute per token is that of a ~50B model while the *knowledge capacity* is that of a 2.8T one. The experts are **latent** — they operate in a learned latent space rather than through hand-tuned token-to-expert heuristics. Scrub a few tokens and watch the selected 16 change: At 1.8% activation, two problems that are mild in a denser MoE become first-order. **Routing:** which 16 experts you pick per token has to be learned well, or the model wastes most of its capacity. **Load balance:** if a few experts hog the tokens, the rest never train, and the effective model collapses to something far smaller than 2.8T. Standard MoEs fight this with an **auxiliary load-balancing loss** and a sensitive balance coefficient — one more hyperparameter that, tuned wrong, either destabilizes training or lets experts collapse. ### Quantile balancing: no auxiliary loss, no knob K3's answer is **Quantile Balancing**: derive each expert's allocation directly from the **quantiles of its router scores**. Set a target quantile — keep the top $q$ fraction of tokens by score — and every expert serves the same fraction of tokens *by construction*, with no auxiliary loss and no balance coefficient to tune. Drag the quantile and flip to the aux-loss regime to see the imbalance it removes: The systems half of the story matters just as much. K3 uses a **fully balanced expert-parallel training method with static shapes and no host synchronization**. That is not a throwaway detail: variable expert loads normally produce variable tensor shapes, which force recompilation and host-side synchronization that stalls a large cluster. Quantile balancing gives every expert the same load, so the shapes are static, so the expert-parallel pipeline runs without host sync — the difference between 16-of-896 routing being a nice idea and being trainable at 2.8T. With all four pieces on the table, here is the paper's module-level picture redrawn: the **Stable LatentMoE** and **KDA** blocks in full detail on the left, and on the right the **Block Attention Residuals** backbone — where each module's output flows through an `α` gate that can read *every* earlier block and the embedding, not just the layer below it. ## Turning compute into intelligence Stack it up — KDA's cheap long-context memory, AttnRes's cheap depth, LatentMoE's extreme-but-stable sparsity, plus refined training and data recipes — and Moonshot's headline is a **~2.5× improvement in overall scaling efficiency** over K2. Concretely: K3 reaches the same capability at roughly **1/2.5 the training compute**. Drag the capability marker: That is the number that actually matters. "2.8 trillion parameters" is a spec-sheet figure; "2.5× more capability per FLOP" is an engineering result. It is what lets an open lab, working around compute limits, ship a frontier-adjacent model without a frontier-sized compute bill. ## What it would take to train it So what does building a 2.8T-A50B model actually cost? The sparsity helps here too: training compute for an MoE scales with the **active** parameters, not the total, so K3's per-token training FLOPs are those of a ~50B model. The standard estimate is $$ C \approx 6 \, N_{\text{active}} \, D $$ with $N_{\text{active}} \approx 50\text{B}$ and $D$ the number of training tokens. Moonshot has not published K3's exact token budget; for reference, K2 was trained on **15.5T tokens** with the Muon optimizer. Plug in a frontier-scale token budget and pick a cluster — the estimate is millions of GPU-hours and weeks of wall-clock: Three things make that estimate *achievable* rather than merely large: - **Per-Head Muon.** K3 extends the Muon optimizer to optimize each attention head independently — more adaptive updates at scale, on top of Muon's already better compute-efficiency than AdamW for this regime. - **MXFP4 / MXFP8 quantization-aware training.** From the SFT stage onward, K3 trains with **MXFP4 weights and MXFP8 activations**. The model is trained to be low-precision-native, which is why the full 2.8T weights fit in roughly **1.4 TB** and why it is servable at all without a quality cliff. - **Static-shape expert parallelism.** As above — quantile balancing plus static shapes and no host synchronization is what keeps a few-thousand-accelerator cluster busy instead of stalling on dynamic routing. The memory reality is worth stating plainly: 2.8T parameters even at 4-bit is ~1.4 TB just for weights, before optimizer states and activations, so training and serving both demand tensor-, pipeline-, and expert-parallelism across many accelerators. This is a systems achievement as much as a modeling one. ## The benchmarks On coding, K3 is a clear #2-or-#3 behind Fable 5 and GPT-5.6 Sol, and ahead of everything else open or closed that Moonshot tested — with a few outright wins.

On FrontierSWE it sits second, close behind Fable 5 and well ahead of the rest: On Terminal Bench 2.1 it is effectively tied for first, and on the long-horizon SWE Marathon and Program Bench it is first outright: The agentic and visual picture is similar — competitive across the board, and #1 on browsing:

The pattern is consistent: K3 wins where the task is long-horizon and tool-heavy (SWE Marathon, Program Bench, BrowseComp, Automation Bench, SpreadsheetBench 2), and comes second to Fable 5 or GPT-5.6 Sol on the single-shot, knowledge-dense ones (GDPval and AA-Briefcase Elo, DeepSWE). For an open model, being in that conversation at all is the story. ## What it costs to serve The sparsity that makes K3 cheap to train makes it cheap to run. API pricing is **$0.30 / MTok** on cache-hit input, **$3.00 / MTok** on cache-miss input, and **$15.00 / MTok** output — and Moonshot reports cache-hit rates **above 90%** in coding workloads, so the effective input price is closer to the cheap number than the expensive one. MXFP4 weights keep the footprint at ~1.4 TB, and Moonshot recommends supernode configurations of **64 or more accelerators**, with a MiniTriton CUDA-core roofline shown on NVIDIA L20. The KDA state and static-shape routing are what make that serving profile hold up at 1M context. **Read the caveats.** (1) K3 still **trails Fable 5 and GPT-5.6 Sol** on overall capability and on user-experience polish — Moonshot says so directly, and flags sensitivity to thinking-history preservation and over-proactiveness in ambiguous situations. (2) The benchmarks are **Moonshot's own suite**, run with opponents' potential fallbacks (Fable 5) and cyberguards (GPT-5.6 Sol) noted on the charts; treat cross-lab numbers as directional. (3) At release the weights were **promised (by 2026-07-27) but not yet downloadable**, so "open" was a commitment, not yet a download. (4) The training cost above is a **first-principles estimate**, not a disclosed figure — Moonshot has not published K3's token budget, cluster, or dollar cost. ## The take Strip away the size record and what is genuinely new in K3 is a coherent set of efficiency bets: **KDA** buys a real 1M context with constant-size memory; **AttnRes** buys depth almost for free; **Stable LatentMoE** with **Quantile Balancing** buys 2.8T of capacity at ~50B of active compute *and* makes that extreme sparsity trainable without an aux-loss knob or host-sync stalls; **Per-Head Muon** and **MXFP4/MXFP8 QAT** make the whole thing converge and fit. The sum is the number that matters — **~2.5× more capability per FLOP than K2** — delivered in the open at 2.8T. It does not top the frontier, and it does not pretend to. What it proves is that the gap between open and closed is now measured in scaling *efficiency*, not in whether an open lab can build at frontier scale at all. --- *Sources: the [Kimi K3 tech blog](https://www.kimi.com/blog/kimi-k3) (architecture, Stable LatentMoE, Quantile Balancing, KDA, AttnRes, benchmarks, pricing) and Moonshot's reported figures. Benchmark numbers are quoted from Moonshot's charts; the training-cost figures are a first-principles estimate from $C \approx 6\,N_{\text{active}}\,D$ with clearly labeled assumptions, using K2's 15.5T-token budget as a reference. Interactive diagrams illustrate the mechanisms; the routing, loop, and cost visuals are illustrative.*