KDA has a half-life: linear attention forgets like a radioactive isotope

2026-08-03 · 8 min · linear-attention · kimi · attention · explainer · math

Here is a small thing I keep turning over. The forgetting mechanism inside Kimi Delta Attention — the linear attention in Kimi K3 — is the same mathematics as radioactive decay. Not "reminiscent of", not "a useful analogy". The same two-line derivation, with tokens where a physicist writes seconds.

Two laws that are one law

A radioactive sample loses a fixed fraction of its remaining atoms per unit time. That gives the exponential law everyone meets in school:

N(t) = N_0 e^{-\lambda t}

A KDA channel loses a fixed fraction of its remaining state per token. Take the recurrence and strip it to the decay term — set the write strength to zero and watch a stored value with no new input arriving:

S_n = \alpha^{\,n} S_0

Those are the same function. Since $\alpha = e^{\ln \alpha}$ ,

\alpha^{\,n} = e^{n \ln \alpha} = e^{-\lambda n}, \qquad \lambda = -\ln \alpha

The retention factor $\alpha$ and the decay constant $\lambda$ are two spellings of one number. A channel with $\alpha$ close to 1 is a long-lived isotope; a channel with small $\alpha$ is one that barely outlives its own creation.

So it has a half-life

Once you accept that, the half-life comes for free. Ask for the $n$ where half the signal is gone:

\alpha^{\,n_{1/2}} = \tfrac{1}{2} \quad\Longrightarrow\quad n_{1/2} \, \ln \alpha = \ln \tfrac{1}{2} \quad\Longrightarrow\quad n_{1/2} = \frac{\ln 0.5}{\ln \alpha}

That is the whole result, and it is worth internalizing because it converts an opaque hyperparameter into a number with units you can reason about. α = 0.99 gives a half-life of about 69 tokens. Not "some decay" — sixty-nine tokens, roughly a long sentence. Drag it:

kda channel decay · half-life in tokens

retention α

0.99974

decay rate λ = −ln α

2.6e-4

half-life n½

2,698

retention factor α0.99974

The curve is Sₙ = αⁿS₀ — a channel’s memory after n tokens. It crosses the dashed half-line at n½ = ln(0.5)/ln(α), the same formula a physicist uses for an isotope, because αⁿ and e^(−λn) are the same function with λ = −ln α. At α = 0.99 a channel has forgotten half of what it knew after about 69 tokens. Nudge α to 0.999 and that becomes roughly 693.

The lever is brutally nonlinear near 1, which is the part worth feeling rather than reading. Going from α = 0.99 to α = 0.999 does not extend memory by a tenth of a percent; it multiplies the half-life by ten, from about 69 tokens to about 693. Each additional nine buys another factor of ten. That is why linear-attention gates are usually parameterized in log space — the useful resolution all lives in the last few decimal places, and a linear parameterization would spend nearly all its range on channels that forget immediately.

The interesting part: α is per channel

If KDA had one global α this would be a cute observation and nothing more. It doesn't. In K3, α is a channel-wise vector — the report writes the state update as

S_t = \left(I - \beta_t k_t k_t^{\top}\right) \mathrm{Diag}(\alpha_t)\, S_{t-1} + \beta_t k_t v_t^{\top}

where $\alpha_t \in (0,1)^{d_k}$ is a per-channel one-step retention factor and $\beta_t$ is the delta-rule write strength. Diag(αₜ) is the load-bearing notation: every one of the $d_k$ channels gets its own decay constant, so a single head carries a whole spectrum of half-lives simultaneously.

one head, many timescales · half-life per channel

1101001k10k100k1M

α = 0.6

1.4 · gone

α = 0.85

4.3 · gone

α = 0.95

14 · gone

α = 0.99

69 · gone

α = 0.997

231 · gone

α = 0.9995

1.4k · gone

α = 0.99993

9.9k · gone

α = 0.999995

139k · holds

α = 0.9999993

990k · holds

context 128K

context length128K tokens

A single α gives a model one memory horizon. Because KDA learns α per channel, one head holds many at once — the fast channels here are effectively a local n-gram window, forgetting within a clause, while the slowest (α = 0.9999993, a half-life near 990k) is still holding half its signal at the end of a million-token context. Slide the context marker and watch the “holds” column collapse from the bottom up as the window grows. That spread is the mechanism behind a linear-attention model having both sharp recency and long recall from the same fixed-size state.

This is what makes a fixed-size state genuinely useful rather than merely cheap. The head is not choosing between "remember recent things sharply" and "remember old things vaguely" — it runs both at once, on different channels. The fast channels behave like a local window: they hold the current clause and dump it. The slow channels are closer to a running summary that survives the entire context. Attention over a KV cache gets its long-range recall by storing everything; KDA gets a version of it by storing a small number of things at deliberately different rates.

It also reframes what "training the gate" means. The model is not learning whether to forget. It is learning a distribution of timescales — effectively allocating channels across memory horizons, the way a filter bank allocates across frequencies.

What K3's config actually pins down

The released weights make a couple of things concrete. K3 runs 69 KDA layers out of 93, three of every four, with a Gated MLA layer as the fourth — so most of the model's sequence mixing is this decay process, and the full-attention layers are the periodic exact-recall anchor. Head dimension is 128, and the gate is full-rank (use_full_rank_gate: true) rather than a low-rank approximation — though see the update below for exactly how the per-channel variation is produced.

The Kimi K3 architecture diagram. On the right, a block repeats a KDA layer paired with a Stable LatentMoE feed-forward network three times, then one Gated MLA layer with its own LatentMoE, with attention-residual weights feeding back from the embedding and preceding blocks. Bottom left, the KDA module expands: linear projections produce q, k, v, alpha and beta, with convolutions and an L2 norm on q and k and sigmoids on the gates, feeding Kimi Delta Attention, a norm, a multiplicative output gate and a final linear. — The three-to-one stack the config describes, drawn: three KDA layers per Gated MLA layer. The bottom-left inset is the KDA module itself, where the per-step retention factor alpha enters as its own projected, sigmoid-gated input alongside the delta-rule write strength beta (Kimi K3 technical report, Figure 2).

The config also carries gate_lower_bound: -5.0. Read as a floor on log-α, that bounds the fastest a channel is allowed to forget: $\alpha \ge e^{-5} \approx 0.0067$ , which is a half-life of about 0.14 tokens — a channel that has essentially dumped its state by the very next step. The ceiling is the interesting end and it is open: as α approaches 1 the half-life grows without bound. To keep half your signal across a full 1M-token context you need α ≈ 0.99999931. That number has seven leading nines, which is exactly why the bound is expressed in log space.

A line plot of the log-decay g against the decay logit z. A grey curve labelled 'Kimi Linear: g = minus e to the A, Softplus of z' falls away without bound toward minus infinity. A red curve labelled 'Kimi K3: g = g-min times Sigmoid of e to the A z' flattens onto a dashed red asymptote at g-min = minus 5. — Why the floor exists and what shape it has: K3 replaces Kimi Linear's unbounded negative-Softplus log-decay with a scaled sigmoid that saturates at g-min = -5, which is the gate_lower_bound in the config and the 0.0067 retention floor in the text (Kimi K3 technical report, Figure 3a).

note

Update, 2026-08-03: the inference above is confirmed. I originally flagged the log-α reading of gate_lower_bound: -5.0 as an assumption, because the config does not state the functional form. It is in the report after all: Eq. 5 defines the per-step log-decay as a scaled sigmoid bounded below by gmin = -5, which is the red curve in Figure 3a above. kimi-k3-in-c, an independent C99 reimplementation, computes the gate exactly that way:

const float a  = expf(A_log[h]);                /* per HEAD  */
const float u  = a * (z[i] + dt_bias[i]);
const float gi = lb * sigmoidf_(u);             /* in (lb, 0]  -> this is log alpha */
alpha[i] = expf(gi);                            /* in (e^lb, 1]                     */

With lb = -5.0, α is bounded to $(e^{-5}, 1] \approx (0.0067, 1]$ — the 0.14-token floor holds.

One refinement the code makes that the config alone did not: A_log is stored per head, and the per-channel variation comes from the z + dt_bias term inside the sigmoid. So a channel's decay is a per-head base rate modulated per channel, rather than a fully independent per-channel parameter. The implementation carries a pointed warning about this — the checkpoint stores head_dim floats but only the first H are nonzero, so indexing A_log per channel is "a silent, fatal error". The Diag(αₜ) structure and the resulting spread of timescales are unaffected.

Why this is more than a nice analogy

Two things fall out of it that are practically useful.

It gives you a unit. "The gate decays the state" is unfalsifiable prose. "This channel has a half-life of 69 tokens" is a claim you can check against a model's behaviour — and it tells you immediately that a channel with a 7-token half-life cannot be the thing carrying a fact across a document, no matter what the attribution heatmap suggests.

It explains the parameterization. Every design choice around these gates — log-space parameterization, bounded gates, careful initialization near 1 — follows from the shape of $n_{1/2} = \ln 0.5 / \ln \alpha$ . The function is nearly flat for most of $(0,1)$ and then explodes in the last sliver. Any scheme that samples α uniformly wastes almost all of its capacity on channels that forget within a few tokens.

The same algebra runs through every gated linear-attention variant, not just KDA — Mamba's $\bar{A}$ , the decay in RetNet and RWKV, the forget gate of an LSTM. They differ in how α is produced and whether it depends on the input. They agree on the underlying law, which has been sitting in physics textbooks the whole time.

Sources: the Kimi K3 technical report for the KDA recurrence, the hybrid layer composition and the two figures reproduced above, and the released Kimi K3 config.json for the layer split, head dimension, use_full_rank_gate and gate_lower_bound. The half-life framing and the derivation are mine; the channel α values in the spectrum widget are illustrative, chosen to span the range, while every half-life shown is computed exactly from them.