# Liquid time constants and gated delta rules: two literatures, one recurrence
> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/ltc-gated-delta
> date: 2026-08-10
> tags: linear-attention, state-space-models, attention, math, explainer, open-source
There are two separate research literatures about neural networks that forget at an input-dependent
rate, and as far as I can tell they mostly do not read each other.
One starts in continuous time. [**Liquid Time-constant
Networks**](https://arxiv.org/abs/2006.04439) (Hasani, Lechner, Amini, Rus and Grosu, 2020) are
ODEs, motivated by the neural dynamics of *C. elegans*, analysed with stability theorems and solved
with numerical integrators. The other starts in discrete time. [**Gated
DeltaNet**](https://arxiv.org/abs/2412.06464) (Yang, Kautz and Hatamizadeh, 2024) is a linear
attention variant, motivated by retrieval failures in efficient Transformers, analysed through
online learning and implemented with chunkwise GPU kernels.
They are the same recurrence. Not analogous — the same. This piece derives that correspondence from
each paper's own equations, works through what each tradition figured out that the other did not,
and then reads [**LTCAttention**](https://github.com/Rikka-Botan/LTCAttention), an implementation
published today that sits deliberately between them.
This is a mechanism piece, so the maths is the point rather than an aside. Everything is derived
from the two papers' numbered equations, and the LTCAttention section is read off its source and its
checked-in result JSON. Where I run a calculation the papers do not — the discretization bridge, and
a scaling estimate at the end — I say so and show the working.
## Part 1 — What "liquid" means
A plain continuous-time RNN decays toward its input at a fixed rate: $dx/dt = -x/\tau + S(t)$, where
$\tau$ is a learned constant. Every input, every timestep, same $\tau$. LTC's move is to let the
decay rate be a function of the state and the input. Substituting
$S(t) = f(\mathbf{x}(t), \mathbf{I}(t), t, \theta)(A - \mathbf{x}(t))$ gives the paper's Equation 1:
$$
\frac{d\mathbf{x}(t)}{dt} = -\left[\frac{1}{\tau} + f(\mathbf{x}(t), \mathbf{I}(t), t, \theta)\right]\mathbf{x}(t) + f(\mathbf{x}(t), \mathbf{I}(t), t, \theta)\,A
$$
Read the bracket. The coefficient multiplying $\mathbf{x}$ is the decay rate, and it now contains
$f$ — a neural network. So the **system time constant** is
$$
\tau_{\text{sys}} = \frac{\tau}{1 + \tau f(\mathbf{x}(t), \mathbf{I}(t), t, \theta)}
$$
which is a number the network computes fresh at every point in time from whatever it is currently
looking at. That is the whole idea, and the name: a time constant that flows.
The paper's two theorems are what make this more than a reparameterization. Because $f$ is a bounded
sigmoidal nonlinearity, **Theorem 1** traps the time constant:
$$
\frac{\tau_i}{1 + \tau_i W_i} \le \tau_{\text{sys}_i} \le \tau_i
$$
and **Theorem 2** traps the state itself between $\min(0, A_i^{\min})$ and $\max(0, A_i^{\max})$,
"which guarantees that the outputs of LTCs never explode even if their inputs grow to infinity."
Those are unusual guarantees. A model whose decay rate is an unconstrained network output could in
principle be driven to instability by an adversarial input; LTC's cannot, by construction.
This is worth flagging because the same argument recurs, unattributed, throughout modern gated linear
attention. Every one of these architectures constrains its gate — Mamba2 through a softplus and a
discretization, Gated DeltaNet by requiring $\alpha_t \in (0,1)$, [Kimi K3's
KDA](/articles/kda-half-life) through a `gate_lower_bound` on $\log\alpha$. The reason is always the
same one LTC proved in 2020: an unbounded forgetting rate is an unbounded system.
## Part 2 — The bridge
Now the part neither literature states, which falls out of LTC's own **Algorithm 1**. Solving
Equation 1 in closed form is not possible, so the paper introduces a *fused solver* — a semi-implicit
Euler step that reads
$$
\mathbf{x}(t + \Delta t) = \frac{\mathbf{x}(t) + \Delta t \, f(\mathbf{x}(t), \mathbf{I}(t), t, \theta) \odot A}{1 + \Delta t\left(\frac{1}{\tau} + f(\mathbf{x}(t), \mathbf{I}(t), t, \theta)\right)}
$$
Look at the denominator. Writing $g$ for the gate output,
$$
1 + \Delta t\left(\tfrac{1}{\tau} + g\right) = 1 + \Delta t\,\frac{1 + \tau g}{\tau} = 1 + \frac{\Delta t}{\tau_{\text{sys}}}
$$
because $\tau_{\text{sys}} = \tau/(1 + \tau g)$ by definition. So the entire update is
$$
\mathbf{x}_{t+1} = \bar{\alpha}\,\mathbf{x}_t + \bar{\alpha}\,\Delta t\, g \odot A,
\qquad
\bar{\alpha} = \frac{1}{1 + \Delta t/\tau_{\text{sys}}}
$$
**That is a gated linear recurrence.** Previous state times a scalar in $(0,1)$, plus a write. The
scalar depends on the input, through $g$. This is structurally identical to what Mamba2, Gated
DeltaNet and KDA do — the object those papers call $\alpha_t$ and describe as a "data-dependent
gating term."
The only difference is which approximation of the exponential you use. The exact solution of the
linear ODE over a step decays by $e^{-\Delta t/\tau_{\text{sys}}}$; LTC's fused solver uses
$1/(1 + \Delta t/\tau_{\text{sys}})$, which is the $[0/1]$ Padé approximant of that exponential.
In the regime these models actually operate in — long memory, so $\Delta t \ll \tau_{\text{sys}}$ —
the two agree to a fraction of a percent. LTC picked the Padé form because it is what makes the
implicit Euler step solvable in closed form; the linear-attention literature picked the exponential
because $\alpha^n$ composes cleanly across a chunk, which is what its parallel scan needs. Same
recurrence, two discretizations, chosen for two different implementation reasons.
Which means the three quantities have one meaning:
| tradition | symbol | this article's other coverage |
|---|---|---|
| continuous-time / LTC | $\tau_{\text{sys}}$, a time constant in seconds | Part 1 above |
| gated linear attention | $\alpha_t = e^{-\Delta t/\tau}$, retention per token | [KDA has a half-life](/articles/kda-half-life) |
| what you should think in | $n_{1/2} = \ln 0.5 / \ln \alpha$, a horizon in tokens | same |
I have argued the third column before, and the LTC connection strengthens it: a half-life is just
$\tau_{\text{sys}}$ in units a language model can be reasoned about in. $\tau$, $\alpha$ and $n_{1/2}$
are one number in three coordinate systems.
## Part 3 — What gating alone cannot do
If the story ended there, Gated DeltaNet would be LTC with better kernels. It is not, and the
difference is the delta rule.
A gated linear attention state is a matrix $\mathbf{S}$ holding key-value associations. Pure gating
updates it as $\mathbf{S}_t = \alpha_t \mathbf{S}_{t-1} + v_t k_t^\top$: scale everything down, add
the new pair. The problem the Gated DeltaNet paper identifies is that $\alpha_t$ is a single number
multiplying the entire state. It can dump everything, and it can hold everything, and it has no way
to express *forget this one fact, keep the rest*.
DeltaNet solved that with the delta rule, which subtracts the state's existing content at the current
key before writing the new one — but, as the paper puts it, "since this process only modifies a
single key-value pair at a time, the model lacks the ability to rapidly clear outdated or irrelevant
information, especially during context switches." One mechanism clears the table but cannot pick up a
single plate; the other picks up single plates but cannot clear the table.
The gated delta rule (Equation 8) is both terms in one product:
$$
\mathbf{S}_t = \mathbf{S}_{t-1}\left(\alpha_t\left(\mathbf{I} - \beta_t k_t k_t^\top\right)\right) + \beta_t v_t k_t^\top
$$
The two bars are the whole argument. Along any direction orthogonal to the current key, the surviving
fraction is $\alpha_t$ — the global forgetting knob. Along $k_t$ itself it is $\alpha_t(1 - \beta_t)$
— global decay *and* targeted erasure. Set $\alpha_t \to 1$ and you have DeltaNet; set $\beta_t \to
0$ and you have Mamba2; the useful region is the interior.
Worth keeping straight against the version of this update I covered in
[KDA has a half-life](/articles/kda-half-life). Kimi K3 writes it as
$S_t = (I - \beta_t k_t k_t^\top)\,\mathrm{Diag}(\alpha_t)\,S_{t-1} + \beta_t k_t v_t^\top$ — the
same two factors, transposed convention, but $\alpha_t$ is a **vector** with one entry per channel
rather than Gated DeltaNet's **scalar** per head. That is not cosmetic. A scalar $\alpha$ gives a
head one memory horizon; a diagonal $\mathrm{Diag}(\alpha)$ gives it a whole spectrum at once, which
is the difference between a head that forgets at one rate and a head that runs a filter bank.
## Part 4 — LTCAttention, and a third place to put a time constant
Both traditions above put the time constant on a **recurrent state**. [LTCAttention by Rikka
Botan](https://github.com/Rikka-Botan/LTCAttention), published today under MIT, puts it somewhere
else: on the attention score itself.
The construction is worth following because it is genuinely clever. Each KV head carries $M$ learned
directions, orthonormalized by QR so that $u_m^\top u_n = \delta_{mn}$. The first token of the causal
block, $x_0$, sets every mode's time constant through one linear projection:
$$
\tau_{h,m}(x_0) = \frac{\tau_{\min}}{\sigma\!\left(r_{h,m} + \delta_{h,m}\right)} > \tau_{\min}
$$
That is the LTC principle exactly — a positive, input-conditioned, *bounded-below* time constant, with
the sigmoid playing the role LTC's Theorem 1 played. Because $x_0$ is visible to every position in the
block, reading it keeps the controller causal.
For a query at $i$ and a key at $j \le i$, with key age $\Delta = i - j$, mode $m$ retains
$\lambda_m(\Delta) = e^{-\Delta/\tau_m}$, and the modes assemble into
$$
M_\Delta(x_0) = \prod_{m=1}^{M}\left[\mathbf{I} - (1 - \lambda_m)u_mu_m^\top\right] = \mathbf{I} + \sum_{m=1}^{M}\left(\lambda_m(\Delta, x_0) - 1\right)u_mu_m^\top
$$
which drops into the score as $s_{ij} = q_i^\top M_{i-j}(x_0)\,k_j/\sqrt{d}$.
The effect: the learned orthogonal complement passes through untouched, while each temporal mode is
an eigenvector with eigenvalue $\lambda_m$. Since $\lambda_m(\Delta) = a_m^\Delta$ with
$a_m = e^{-1/\tau_m}$, this is the same stable diagonal decay law as an SSM — just expressed as a
metric on an inner product rather than a state update.
### The factorization is the load-bearing trick, and it checks out
Applying a different $M_\Delta$ to every $(i,j)$ pair naively means building a $T \times T \times d$
object. LTCAttention avoids it by pushing the decay into the queries and keys separately, around a
fixed center $c$:
$$
q_i' = q_i + \sum_m \left(e^{-\frac{i-c}{\tau_m}} - 1\right)(q_i^\top u_m)u_m,
\qquad
k_j' = k_j + \sum_m \left(e^{\frac{j-c}{\tau_m}} - 1\right)(k_j^\top u_m)u_m
$$
I checked the algebra rather than taking it on faith. Decompose $q_i = q_\perp + \sum_m (q_i^\top
u_m)u_m$ using orthonormality; the transform replaces each modal coefficient by
$e^{-(i-c)/\tau_m}(q_i^\top u_m)$ and leaves $q_\perp$ alone, and symmetrically for $k$. Their inner
product is then
$$
q_i'^\top k_j' = q_\perp^\top k_\perp + \sum_m e^{-\frac{i-c}{\tau_m}}e^{\frac{j-c}{\tau_m}}(q_i^\top u_m)(k_j^\top u_m) = q_\perp^\top k_\perp + \sum_m e^{-\frac{i-j}{\tau_m}}(q_i^\top u_m)(k_j^\top u_m)
$$
and expanding $q_i^\top M_\Delta k_j$ directly gives the same thing. The center $c$ cancels, exactly
as claimed. The modal projections cost $O(TMd)$, so **scaled dot-product attention remains the only
quadratic operation** — the mechanism is free at the asymptotic level and the standard SDPA kernel is
still doing the heavy lifting.
There is a real numerical hazard hiding in that trick, and the code knows it. The factors
$e^{-(i-c)/\tau}$ and $e^{(j-c)/\tau}$ are individually huge or tiny even though their product is
bounded by 1; they cancel only algebraically. The implementation handles this two ways. It computes
the exponents in FP32 or FP64 regardless of the BF16 activation dtype, with a comment saying exactly
why. And it fixes $c$ at the middle of the context, `centre = 0.5 * (max_positions - 1)`, rather than
recomputing it per prefix — which both keeps cached keys valid as the KV cache grows and halves the
worst-case exponent.
The choice of $\tau_{\min}$ then finishes the job, and this is my favourite detail in the repository.
The default is `min_tau = max_positions / 12`. Combined with the centered origin, the largest
exponent magnitude is
$$
\frac{(T-1)/2}{T/12} = \frac{6(T-1)}{T} \approx 6
$$
**independent of context length.** Whatever $T$ you configure, the factorization's intermediate values
stay inside roughly $e^{\pm 6}$. That is not a coincidence; it is a bound chosen so the trick cannot
overflow.
### The experiment, and the number that worries me
The repository ships a real controlled study rather than a claim: three seeds, a paired comparison,
SHA-256 checksums on the tokenized data, one epoch over 287,588,352 FineWeb-Edu tokens consumed
without replacement, and the full result JSON checked in.
Reading the numbers straight out of `results/fineweb_edu_fullrank_29m_6layer_half_3seeds.json`:
| | validation loss | perplexity |
|---|---|---|
| standard | 4.13471 ± 0.02367 | 62.49 |
| LTC | 4.07281 ± 0.01777 | 58.73 |
| paired difference | **−0.06190 ± 0.00646** | |
The per-seed differences are −0.0544, −0.0702 and −0.0610 — negative in all three, with a spread ten
times smaller than the effect. As a paired result at this scale that is about as clean as three seeds
get, and the README is careful to say that "three seeds and one small model scale do not establish
broad scaling behavior."
Two confounds are worth quantifying, and the repository reports exactly the numbers needed to do it.
**Parameters.** LTC adds 345,600 of them, +1.20%. Borrowing the Chinchilla-form sensitivity
$\partial L \approx \alpha\,(A/N^\alpha)\,(\partial N/N)$ with $\alpha = 0.34$, a 1.20% parameter
increase at 28.8M is worth roughly **0.005 nats**. The observed effect is more than ten times that.
The gain is not just parameter count.
**Compute.** This is the one. LTC also runs **12.40% slower** (142,473 vs 162,638 tokens/sec, measured
and reported by the author). The comparison is token-matched, not wall-clock-matched. Spend that same
12.4% on more training tokens for the baseline instead, and the same scaling form
($\beta = 0.28$ on the data term) predicts a gain of roughly **0.061 nats** — which is, to two
decimal places, the entire measured effect.
I want to be precise about what that estimate is and is not. The coefficients come from a scaling law
fitted on a different corpus, tokenizer and budget, so the *absolute* numbers do not transfer; I am
borrowing only the sensitivity, and the error bars on that are wide. The near-exact agreement between
0.061 and 0.062 is a coincidence of a rough calculation, not a measurement. But the direction is
robust: at this scale, a 12% throughput penalty buys enough extra tokens to be the same order as the
observed quality gain. **The missing experiment is a wall-clock-matched run**, and until someone does
it the honest reading is that LTCAttention is better per token and undetermined per second. That the
author reported the throughput cost at all is what makes this check possible — most releases do not.
One further limitation, stated plainly in the repo: the released code is LTC-only, and the baseline
artifacts are "retained only as experiment provenance." So the comparison cannot currently be re-run
from this repository, only re-read.
## What each tradition knows
Setting the implementations aside, the two literatures have complementary blind spots.
**LTC knows about stability and it knows about time.** It has proofs that the time constant and the
state stay bounded under arbitrary input. It treats $\Delta t$ as a real quantity, which means it
handles irregularly sampled sequences natively — a capability the discrete-time literature mostly
gave up without noticing, because tokens arrive on a uniform grid. And it thinks in a unit, seconds,
that forces you to ask how long a memory is supposed to last.
**Gated linear attention knows about scale and it knows about writing.** It has the chunkwise parallel
algorithms that make these recurrences trainable on modern hardware at all, which is the entire reason
the idea reached billion-parameter models. And it has the delta rule — a way to modify one association
without disturbing the others that has no counterpart in the LTC formulation, where the "write" is
just $f \cdot A$ added to a decaying state.
LTCAttention is interesting mostly as evidence that the gap is crossable in either direction: it takes
LTC's bounded input-conditioned $\tau$, GDN's adaptive retention, and applies them to a third
substrate neither paper considered. Whether that particular hybrid pays for its 12% is, on the
evidence available, not yet settled. Whether the two literatures should be reading each other seems
to me much clearer.
---
*Sources: [Liquid Time-constant Networks](https://arxiv.org/abs/2006.04439) (arXiv 2006.04439,
Hasani, Lechner, Amini, Rus, Grosu) for Equation 1, Algorithm 1, and Theorems 1–2, read via ar5iv;
[Gated Delta Networks: Improving Mamba2 with Delta Rule](https://arxiv.org/abs/2412.06464) (arXiv
2412.06464, Yang, Kautz, Hatamizadeh) for Equation 8 and the complementarity argument; and the
[LTCAttention repository](https://github.com/Rikka-Botan/LTCAttention) at its 2026-08-10 state —
`README.md`, `model.py`, `config/`, and `results/fineweb_edu_fullrank_29m_6layer_half_3seeds.json`.
The three figures are LTCAttention's own, flattened onto white. The fused-solver-to-gated-recurrence
derivation, the verification of the query-key factorization, the $\tau_{\min} = T/12$ bound, and both
scaling estimates are mine and are shown in full above so they can be checked. All four interactives
are mine.*