# KDA has a half-life: linear attention forgets like a radioactive isotope
> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/kda-half-life
> date: 2026-08-03
> tags: linear-attention, kimi, attention, explainer, math
Here is a small thing I keep turning over. The forgetting mechanism inside **Kimi Delta Attention** — the linear
attention in [Kimi K3](/articles/kimi-k3) — is *the same mathematics as radioactive decay*. Not "reminiscent of",
not "a useful analogy". The same two-line derivation, with tokens where a physicist writes seconds.
## Two laws that are one law
A radioactive sample loses a fixed *fraction* of its remaining atoms per unit time. That gives the exponential
law everyone meets in school:
$$
N(t) = N_0 e^{-\lambda t}
$$
A KDA channel loses a fixed *fraction* of its remaining state per token. Take the recurrence and strip it to the
decay term — set the write strength to zero and watch a stored value with no new input arriving:
$$
S_n = \alpha^{\,n} S_0
$$
Those are the same function. Since $\alpha = e^{\ln \alpha}$,
$$
\alpha^{\,n} = e^{n \ln \alpha} = e^{-\lambda n}, \qquad \lambda = -\ln \alpha
$$
The retention factor $\alpha$ and the decay constant $\lambda$ are two spellings of one number. A channel with
$\alpha$ close to 1 is a long-lived isotope; a channel with small $\alpha$ is one that barely outlives its own
creation.
## So it has a half-life
Once you accept that, the half-life comes for free. Ask for the $n$ where half the signal is gone:
$$
\alpha^{\,n_{1/2}} = \tfrac{1}{2}
\quad\Longrightarrow\quad
n_{1/2} \, \ln \alpha = \ln \tfrac{1}{2}
\quad\Longrightarrow\quad
n_{1/2} = \frac{\ln 0.5}{\ln \alpha}
$$
That is the whole result, and it is worth internalizing because it converts an opaque hyperparameter into a
number with units you can reason about. **α = 0.99 gives a half-life of about 69 tokens.** Not "some decay" —
sixty-nine tokens, roughly a long sentence. Drag it:
The lever is brutally nonlinear near 1, which is the part worth feeling rather than reading. Going from
α = 0.99 to α = 0.999 does not extend memory by a tenth of a percent; it multiplies the half-life by ten, from
about 69 tokens to about 693. Each additional nine buys another factor of ten. That is why linear-attention
gates are usually parameterized in log space — the useful resolution all lives in the last few decimal places,
and a linear parameterization would spend nearly all its range on channels that forget immediately.
A useful sanity check: half-life is a property of the *ratio*, not the magnitude. A channel at α = 0.99 has lost
half its signal after 69 tokens, three quarters after 138, and about a thousandth of it survives to 690 — ten
half-lives, the same "ten half-lives and it's gone" rule of thumb used for isotopes.
## The interesting part: α is per channel
If KDA had one global α this would be a cute observation and nothing more. It doesn't. In K3, α is a
**channel-wise** vector — the report writes the state update as
$$
S_t = \left(I - \beta_t k_t k_t^{\top}\right) \mathrm{Diag}(\alpha_t)\, S_{t-1} + \beta_t k_t v_t^{\top}
$$
where $\alpha_t \in (0,1)^{d_k}$ is a **per-channel** one-step retention factor and $\beta_t$ is the delta-rule
write strength. `Diag(αₜ)` is the load-bearing notation: every one of the $d_k$ channels gets its own decay
constant, so a single head carries a whole spectrum of half-lives simultaneously.
This is what makes a fixed-size state genuinely useful rather than merely cheap. The head is not choosing
between "remember recent things sharply" and "remember old things vaguely" — it runs both at once, on different
channels. The fast channels behave like a local window: they hold the current clause and dump it. The slow
channels are closer to a running summary that survives the entire context. Attention over a KV cache gets its
long-range recall by *storing everything*; KDA gets a version of it by storing a small number of things at
deliberately different rates.
It also reframes what "training the gate" means. The model is not learning *whether* to forget. It is learning a
distribution of timescales — effectively allocating channels across memory horizons, the way a filter bank
allocates across frequencies.
## What K3's config actually pins down
The released weights make a couple of things concrete. K3 runs **69 KDA layers out of 93**, three of every four,
with a Gated MLA layer as the fourth — so most of the model's sequence mixing is this decay process, and the
full-attention layers are the periodic exact-recall anchor. Head dimension is 128, and the gate is full-rank
(`use_full_rank_gate: true`) rather than a low-rank approximation — though see the update below for exactly how
the per-channel variation is produced.
The config also carries `gate_lower_bound: -5.0`. Read as a floor on log-α, that bounds the fastest a channel is
allowed to forget: $\alpha \ge e^{-5} \approx 0.0067$, which is a half-life of about **0.14 tokens** — a channel
that has essentially dumped its state by the very next step. The ceiling is the interesting end and it is open:
as α approaches 1 the half-life grows without bound. To keep half your signal across a full 1M-token context you
need α ≈ 0.99999931. That number has seven leading nines, which is exactly why the bound is expressed in log
space.
**Update, 2026-08-03: the inference above is confirmed.** I originally flagged the log-α reading of
`gate_lower_bound: -5.0` as an assumption — the config does not state the functional form, and the report does not
either. [kimi-k3-in-c](/articles/kimi-k3-in-c), an independent C99 reimplementation, computes the gate exactly
that way:
```c
const float a = expf(A_log[h]); /* per HEAD */
const float u = a * (z[i] + dt_bias[i]);
const float gi = lb * sigmoidf_(u); /* in (lb, 0] -> this is log alpha */
alpha[i] = expf(gi); /* in (e^lb, 1] */
```
With `lb = -5.0`, α is bounded to $(e^{-5}, 1] \approx (0.0067, 1]$ — the 0.14-token floor holds.
One refinement the code makes that the config alone did not: `A_log` is stored **per head**, and the per-channel
variation comes from the `z + dt_bias` term inside the sigmoid. So a channel's decay is a per-head base rate
modulated per channel, rather than a fully independent per-channel parameter. The implementation carries a pointed
warning about this — the checkpoint stores `head_dim` floats but only the first `H` are nonzero, so indexing
`A_log` per channel is *"a silent, fatal error"*. The `Diag(αₜ)` structure and the resulting spread of timescales
are unaffected.
## Why this is more than a nice analogy
Two things fall out of it that are practically useful.
**It gives you a unit.** "The gate decays the state" is unfalsifiable prose. "This channel has a half-life of 69
tokens" is a claim you can check against a model's behaviour — and it tells you immediately that a channel with a
7-token half-life cannot be the thing carrying a fact across a document, no matter what the attribution heatmap
suggests.
**It explains the parameterization.** Every design choice around these gates — log-space parameterization,
bounded gates, careful initialization near 1 — follows from the shape of $n_{1/2} = \ln 0.5 / \ln \alpha$. The
function is nearly flat for most of $(0,1)$ and then explodes in the last sliver. Any scheme that samples α
uniformly wastes almost all of its capacity on channels that forget within a few tokens.
The same algebra runs through every gated linear-attention variant, not just KDA — Mamba's $\bar{A}$, the decay
in RetNet and RWKV, the forget gate of an LSTM. They differ in how α is produced and whether it depends on the
input. They agree on the underlying law, which has been sitting in physics textbooks the whole time.
---
*Sources: the [Kimi K3 technical report](https://github.com/MoonshotAI/Kimi-K3/blob/main/k3_tech_report.pdf) for
the KDA recurrence and the hybrid layer composition, and the released
[Kimi K3 `config.json`](https://huggingface.co/moonshotai/Kimi-K3) for the layer split, head dimension,
`use_full_rank_gate` and `gate_lower_bound`. The half-life framing and the derivation are mine; the channel
α values in the spectrum widget are illustrative, chosen to span the range, while every half-life shown is
computed exactly from them.*