KDA has a half-life: linear attention forgets like a radioactive isotope

2026-08-03 · 7 min · linear-attention · kimi · attention · explainer · math

Here is a small thing I keep turning over. The forgetting mechanism inside Kimi Delta Attention — the linear attention in Kimi K3 — is the same mathematics as radioactive decay. Not "reminiscent of", not "a useful analogy". The same two-line derivation, with tokens where a physicist writes seconds.

Two laws that are one law

A radioactive sample loses a fixed fraction of its remaining atoms per unit time. That gives the exponential law everyone meets in school:

N(t) = N_0 e^{-\lambda t}

A KDA channel loses a fixed fraction of its remaining state per token. Take the recurrence and strip it to the decay term — set the write strength to zero and watch a stored value with no new input arriving:

S_n = \alpha^{\,n} S_0

Those are the same function. Since $\alpha = e^{\ln \alpha}$ ,

\alpha^{\,n} = e^{n \ln \alpha} = e^{-\lambda n}, \qquad \lambda = -\ln \alpha

The retention factor $\alpha$ and the decay constant $\lambda$ are two spellings of one number. A channel with $\alpha$ close to 1 is a long-lived isotope; a channel with small $\alpha$ is one that barely outlives its own creation.

So it has a half-life

Once you accept that, the half-life comes for free. Ask for the $n$ where half the signal is gone:

\alpha^{\,n_{1/2}} = \tfrac{1}{2} \quad\Longrightarrow\quad n_{1/2} \, \ln \alpha = \ln \tfrac{1}{2} \quad\Longrightarrow\quad n_{1/2} = \frac{\ln 0.5}{\ln \alpha}

That is the whole result, and it is worth internalizing because it converts an opaque hyperparameter into a number with units you can reason about. α = 0.99 gives a half-life of about 69 tokens. Not "some decay" — sixty-nine tokens, roughly a long sentence. Drag it:

kda channel decay · half-life in tokens

retention α

0.99974

decay rate λ = −ln α

2.6e-4

half-life n½

2,698

retention factor α0.99974

The curve is Sₙ = αⁿS₀ — a channel’s memory after n tokens. It crosses the dashed half-line at n½ = ln(0.5)/ln(α), the same formula a physicist uses for an isotope, because αⁿ and e^(−λn) are the same function with λ = −ln α. At α = 0.99 a channel has forgotten half of what it knew after about 69 tokens. Nudge α to 0.999 and that becomes roughly 693.

The lever is brutally nonlinear near 1, which is the part worth feeling rather than reading. Going from α = 0.99 to α = 0.999 does not extend memory by a tenth of a percent; it multiplies the half-life by ten, from about 69 tokens to about 693. Each additional nine buys another factor of ten. That is why linear-attention gates are usually parameterized in log space — the useful resolution all lives in the last few decimal places, and a linear parameterization would spend nearly all its range on channels that forget immediately.

The interesting part: α is per channel

If KDA had one global α this would be a cute observation and nothing more. It doesn't. In K3, α is a channel-wise vector — the report writes the state update as

S_t = \left(I - \beta_t k_t k_t^{\top}\right) \mathrm{Diag}(\alpha_t)\, S_{t-1} + \beta_t k_t v_t^{\top}

where $\alpha_t \in (0,1)^{d_k}$ is a per-channel one-step retention factor and $\beta_t$ is the delta-rule write strength. Diag(αₜ) is the load-bearing notation: every one of the $d_k$ channels gets its own decay constant, so a single head carries a whole spectrum of half-lives simultaneously.

one head, many timescales · half-life per channel

1101001k10k100k1M

α = 0.6

1.4 · gone

α = 0.85

4.3 · gone

α = 0.95

14 · gone

α = 0.99

69 · gone

α = 0.997

231 · gone

α = 0.9995

1.4k · gone

α = 0.99993

9.9k · gone

α = 0.999995

139k · holds

α = 0.9999993

990k · holds

context 128K

context length128K tokens

A single α gives a model one memory horizon. Because KDA learns α per channel, one head holds many at once — the fast channels here are effectively a local n-gram window, forgetting within a clause, while the slowest (α = 0.9999993, a half-life near 990k) is still holding half its signal at the end of a million-token context. Slide the context marker and watch the “holds” column collapse from the bottom up as the window grows. That spread is the mechanism behind a linear-attention model having both sharp recency and long recall from the same fixed-size state.

This is what makes a fixed-size state genuinely useful rather than merely cheap. The head is not choosing between "remember recent things sharply" and "remember old things vaguely" — it runs both at once, on different channels. The fast channels behave like a local window: they hold the current clause and dump it. The slow channels are closer to a running summary that survives the entire context. Attention over a KV cache gets its long-range recall by storing everything; KDA gets a version of it by storing a small number of things at deliberately different rates.

It also reframes what "training the gate" means. The model is not learning whether to forget. It is learning a distribution of timescales — effectively allocating channels across memory horizons, the way a filter bank allocates across frequencies.

What K3's config actually pins down

The released weights make a couple of things concrete. K3 runs 69 KDA layers out of 93, three of every four, with a Gated MLA layer as the fourth — so most of the model's sequence mixing is this decay process, and the full-attention layers are the periodic exact-recall anchor. Head dimension is 128, and the gate is full-rank (use_full_rank_gate: true) rather than a low-rank approximation — though see the update below for exactly how the per-channel variation is produced.

The config also carries gate_lower_bound: -5.0. Read as a floor on log-α, that bounds the fastest a channel is allowed to forget: $\alpha \ge e^{-5} \approx 0.0067$ , which is a half-life of about 0.14 tokens — a channel that has essentially dumped its state by the very next step. The ceiling is the interesting end and it is open: as α approaches 1 the half-life grows without bound. To keep half your signal across a full 1M-token context you need α ≈ 0.99999931. That number has seven leading nines, which is exactly why the bound is expressed in log space.

note

Update, 2026-08-03: the inference above is confirmed. I originally flagged the log-α reading of gate_lower_bound: -5.0 as an assumption — the config does not state the functional form, and the report does not either. kimi-k3-in-c, an independent C99 reimplementation, computes the gate exactly that way:

const float a  = expf(A_log[h]);                /* per HEAD  */
const float u  = a * (z[i] + dt_bias[i]);
const float gi = lb * sigmoidf_(u);             /* in (lb, 0]  -> this is log alpha */
alpha[i] = expf(gi);                            /* in (e^lb, 1]                     */

With lb = -5.0, α is bounded to $(e^{-5}, 1] \approx (0.0067, 1]$ — the 0.14-token floor holds.

One refinement the code makes that the config alone did not: A_log is stored per head, and the per-channel variation comes from the z + dt_bias term inside the sigmoid. So a channel's decay is a per-head base rate modulated per channel, rather than a fully independent per-channel parameter. The implementation carries a pointed warning about this — the checkpoint stores head_dim floats but only the first H are nonzero, so indexing A_log per channel is "a silent, fatal error". The Diag(αₜ) structure and the resulting spread of timescales are unaffected.

Why this is more than a nice analogy

Two things fall out of it that are practically useful.

It gives you a unit. "The gate decays the state" is unfalsifiable prose. "This channel has a half-life of 69 tokens" is a claim you can check against a model's behaviour — and it tells you immediately that a channel with a 7-token half-life cannot be the thing carrying a fact across a document, no matter what the attribution heatmap suggests.

It explains the parameterization. Every design choice around these gates — log-space parameterization, bounded gates, careful initialization near 1 — follows from the shape of $n_{1/2} = \ln 0.5 / \ln \alpha$ . The function is nearly flat for most of $(0,1)$ and then explodes in the last sliver. Any scheme that samples α uniformly wastes almost all of its capacity on channels that forget within a few tokens.

The same algebra runs through every gated linear-attention variant, not just KDA — Mamba's $\bar{A}$ , the decay in RetNet and RWKV, the forget gate of an LSTM. They differ in how α is produced and whether it depends on the input. They agree on the underlying law, which has been sitting in physics textbooks the whole time.

Sources: the Kimi K3 technical report for the KDA recurrence and the hybrid layer composition, and the released Kimi K3 config.json for the layer split, head dimension, use_full_rank_gate and gate_lower_bound. The half-life framing and the derivation are mine; the channel α values in the spectrum widget are illustrative, chosen to span the range, while every half-life shown is computed exactly from them.