Ling-3.0-flash: a 124B open MoE that runs like a 5B and reaches for 1M tokens

Ling-3.0-flash, released 2026-07-23 by inclusionAI (Ant Group's Ling/Bailing MoE family), is a 124B-parameter open Mixture-of-Experts that activates only ~5.1B parameters per token — about 4%. It ships with a 256K native context that is designed to extend to 1M tokens, a hybrid-reasoning ("thinking") mode, and a claim that lands harder than the spec sheet: with roughly 1/8 the total and 1/12 the active parameters, it matches or beats inclusionAI's own 1T flagship on most of the benchmarks it was launched against.

The interesting part is not the sparsity ratio on its own — it's the attention stack that makes a cheap million-token context tractable. Ling-3.0-flash is built on hybrid linear attention: it interleaves KDA (Kimi Delta Attention) — a gated delta-rule linear-attention layer with constant-size memory — with Gated MLA full attention, at a 5:1 ratio, and feeds each block into a 512-expert MoE that fires just 8 experts plus 1 shared expert per token. This is a first-principles tour of each piece, why it matters, and where the launch numbers put it.

Read the center stack bottom-to-top: tokens are embedded, pass through 7 groups of blocks — each group is 5× (KDA + MoE) followed by 1× (Gated MLA + MoE) — then a final norm, a 157k-vocab output projection, and a multi-token-prediction head. The two right-hand panels expand the attention modules. Take the ideas one at a time.

KDA: constant-size memory over a very long context

Ordinary softmax attention keeps a KV cache that grows by one entry per token. At a 256K–1M context that cache is the cost: decoding becomes memory-bound on a cache that scales with sequence length, and it only gets heavier as the context fills.

KDA avoids that. It is a gated delta-rule linear attention: instead of a growing cache it keeps a fixed-size recurrent state $S_t$ that each token updates in place — it erases a little of the old state (a per-channel gated decay) and writes the new key/value association (the delta rule). A compact way to write the family is

where $\alpha_t$ is the gated decay (the erase), $\beta_t\, k_t v_t^{\top}$ is the written delta, and $o_t$ reads the state with the query. The state $S_t$ is a fixed $d \times d$ matrix — its size does not depend on how many tokens came before. That is exactly the structure the right-hand KDA panel draws: queries and keys go through a short conv and an L2 norm, values through a conv, and learned $\alpha$/$\beta$ gates (softplus $\varphi$ and sigmoid $\sigma$) control the decay and write, with a final sigmoid output gate.

Because the state is constant-size, KDA runs in linear time and constant memory in the sequence length — so five of every six layers pay no growing-cache cost at all. Only the single Gated-MLA layer per group keeps a real KV cache, so the model's total long-context memory grows at roughly 1/6 the slope of an all-attention model. Drag the context length and watch the gap open up:

This is the lever behind "1M-token context" being a design target rather than a marketing number. It is not free — a linear-attention state is a lossy summary, not a perfect record — which is exactly why Ling keeps full attention in the mix.

Gated MLA: one full-attention layer per group for exact recall

Every group's sixth block is a Gated MLA layer — Multi-head Latent Attention with RoPE and a learned sigmoid gate. MLA compresses keys and values into a low-rank latent before attending, which shrinks the KV cache of the full-attention layers themselves; RoPE gives the positional structure that makes long-range exact recall work. The left panel's callout says it plainly: the 1M-token context rides on RoPE-equipped MLA, while KDA carries the linear time complexity.

The division of labour is the whole point. KDA is cheap but forgetful; full attention is exact but expensive. A 5:1 interleave keeps one exact-recall layer in every group so the lossy linear layers have something precise to anchor to — you get most of full attention's fidelity at a fraction of its memory. This is the same bet Moonshot made in Kimi K3, and the resemblance is not subtle (more on that below).

The MoE: 8 of 512, plus a shared expert, kept balanced

Ling-3.0-flash's feed-forward is a fine-grained MoE: 512 routed experts, of which only 8 fire per token — an activation ratio of 1/64 — plus 1 shared expert that runs on every token to carry the common, always-useful computation. That extreme sparsity is what lets a 124B model spend only ~5.1B parameters per token: the compute is that of a ~5B model while the knowledge capacity is that of a 124B one.

At 1/64 activation two problems that are mild in a denser MoE turn first-order. Routing has to be learned well or most of the capacity is wasted, and load balance matters even more — if a few experts hog the tokens, the rest never train and the effective model collapses to something far smaller than 124B. Ling's answer is ALF-LB (adaptive load balancing): rather than a single brittle auxiliary-loss coefficient, it adapts the balancing pressure per expert so utilisation stays even without destabilising training — the same spirit as K3's aux-loss-free balancing, aimed at keeping all 512 experts alive.

Two more structural details worth stating:

• The first 2 blocks use a dense FFN instead of MoE. Early layers do broad, low-level feature mixing where routing buys little and can hurt stability, so Ling keeps them dense and only switches to sparse experts deeper in the stack — a common, load-bearing choice at this sparsity.
• Multi-Token Prediction (MTP). The training objective is next-token prediction plus an auxiliary multi-token-prediction head (the node at the top of the stack). MTP densifies the learning signal per step and doubles as a self-speculative decoding draft head at inference, which is part of how a "flash" model earns the name.

Rounding out the sheet: a 157k-token vocabulary, an embedding dimension of 2,560, and the 7-group hybrid stack above.

The Kimi K3 resemblance

If this all sounds familiar, it should. Kimi K3 is built on the same three bets — KDA for constant-size long-context memory, full attention interleaved for exact recall, and an extreme-but-stable sparse MoE with a balancing scheme that avoids a brittle aux-loss knob. Ling-3.0-flash runs the same playbook at a very different scale: 124B/5.1B for a fast production model versus K3's 2.8T/~50B frontier system, and 8-of-512 routing versus K3's 16-of-896. The convergence is the story — two independent open labs arriving at the same architecture for efficient long-context reasoning strongly suggests this hybrid-linear + sparse-MoE recipe is where open models are settling.

The benchmarks

On its launch suite, inclusionAI compares the Ling-3.0-flash(RC3)-Thinking build against a field of thinking models: its own 1T Ring-2.6-1T, MiniMax-M2.7, Step-3.7-Flash-high, Deepseek-v4-flash-max, Nemotron-3-Super-120B, GPT-5.4-mini-high, and Claude-Sonnet-4.6-maxthink. The full grid:

expand

The headline result is coding. On SWE-Bench Pro the 124B model edges the entire field — including the 1T sibling and every frontier opponent it was tested against:

Long-context and instruction-following are where the hybrid stack should pay off, and it does. On MRCR-128k it sits second, just behind Claude and comfortably ahead of the 1T Ring and everything else — while GPT-5.4-mini, MiniMax and Step fall off a cliff:

On SysBench (system-prompt adherence) it's in a three-way near-tie at the top with Claude and Deepseek:

It is not a clean sweep. On Terminal-Bench v2.1-AA — long-horizon, tool-heavy agent work — Claude-Sonnet-4.6 is well clear and Deepseek leads the open pack; Ling lands mid-field, ahead of its own 1T sibling but not the frontier:

The pattern is consistent with the architecture: Ling is strongest where structured recall and instruction adherence dominate (SWE-Bench Pro, MRCR-128k, SysBench, IFBench), and merely competitive on the longest-horizon agent loops where a top proprietary model still pulls ahead. For a 124B model activating ~5.1B parameters, being in that conversation — and beating a 1T model at 1/12 the active compute — is the result.

The take

Strip away the "beats a 1T model" headline and what's genuinely useful about Ling-3.0-flash is a coherent, reproducible recipe: KDA buys a linear-time, constant-memory path to very long context; a 1-in-6 Gated MLA layer buys back the exact recall linear attention loses; an 8-of-512 + shared-expert MoE with ALF-LB buys 124B of capacity at ~5B of active compute and keeps all the experts trained; and MTP plus dense early blocks make the whole thing converge and decode fast. That it is essentially the Kimi K3 architecture at 1/22 the size is the most telling part — the frontier recipe for efficient long-context reasoning is now open, and it runs on a single node.

Sources: the Ling-3.0-flash model card and inclusionAI's launch materials (architecture, hybrid KDA/MLA interleave, MoE configuration, benchmarks), the Kilo announcement (124B/5.1B, 256K→1M context), and inclusionAI's launch benchmark chart (reproduced above). Benchmark numbers are quoted from inclusionAI's own report for the RC3-Thinking build. The architecture diagram is a house-style recreation of the launch figure; the KV-vs-state chart is illustrative (order-of-magnitude, to show the shape).