2026-08-14 · 7 min · transformers · architecture · reasoning · paper · efficiency · explainer
Full-bandwidth transformer (arXiv 2608.08888, 2026-08-09) opens with an observation that is obvious once stated and easy to never state:
Autoregressive transformers compute along two axes: horizontally across generated tokens, and vertically through model depth. Dense attention gives each token broad horizontal access to the past, but the vertical feedback channel between decoding steps remains narrow: only the sampled token returns to the bottom of the stack, while the top-layer hidden state is discarded.
Every decoding step runs the full depth of the model, produces a rich final-layer state, uses it to pick one token from the vocabulary — and then throws the state away. The next step starts from that one token.
How narrow is narrow
the sampled token is the only thing a standard transformer carries forward; the top-layer state is discarded at every step
Be careful with the ratio: a hidden state’s float width is an upper bound on what it could carry, not a measurement of what it does. Nobody is claiming 24 kilobits of useful signal per step. The claim that holds is about the channel’s shape — one discrete symbol drawn from a fixed alphabet, versus a continuous vector — and about what that shape costs. A model whose only way to pass intermediate state forward is to name it in the vocabulary has to verbalize its own scratch work, which is a fair description of what chain-of-thought is.
The paper counts it in bits: a sampled token carries log₂|V| bits between steps. Their model has a tied 100,352-token vocabulary, so 16.6 bits per step, against a 1,536-dimensional hidden state.
I would not push the ratio too hard — a hidden state's float width bounds what it could carry, not what it does — and the paper doesn't either. The claim that holds is about the channel's shape: one discrete symbol from a fixed alphabet, versus a continuous vector.
That shape has a consequence, and it reframes something familiar. If the only way to pass intermediate state to your next step is to name it in the vocabulary, then you must verbalize your own scratch work. Which is a fair description of what chain-of-thought is:
CoT sidesteps this by externalizing intermediate state into language: the model writes out partial results, subgoals, and bookkeeping, then conditions future computation on the written trace.
Reasoning traces on this reading are not primarily a thinking technique. They are a workaround for a 16-bit bus.
The fix
Latent feedback. At each decoding step, fuse the previous top-layer hidden state with the sampled token's embedding through a gated linear unit, and use that as the next input. The state carried between steps goes from s_t = a_{1:t} — the token trace alone — to s_t = (a_{1:t}, z_t), the trace and the most recent latent.
What I find persuasive is what it does not change. Standard transformer architecture, standard KV cache, standard language-modelling objective. The fusion is dimension-preserving, so nothing downstream needs to know. There is an appendix on vLLM compatibility.
The paper's phrase for the benefit is the right one: latent feedback lets "non-verbalized computation re-enter the stack with a renewed depth budget." A fixed-depth transformer has bounded serial computation per forward pass; feeding the top state back gives the next pass somewhere to continue from rather than somewhere to restart.
Training is the part that could have gone wrong. Naive recurrence destroys parallel teacher forcing and with it the ability to train at scale. Their answer is a scheduled multi-pass objective: introduce latent feedback late in pretraining, and mix in a small fraction of deeper feedback passes for stability.
What it buys
None stated — and this is the version that costs nothing to deploy, since the serving pipeline is unchanged. If only one thing from this paper survives, it should be this.
The conciseness result is the one I keep turning over. It appears on the base model and vanishes after instruction tuning, and the paper’s explanation is that the tuning traces were written by a model that had no choice but to verbalize. If that is right, then a capability can be trained in and then trained straight back out by imitation data that predates it — which is a problem that generalizes well beyond this architecture.
At 1B parameters and up to 400B tokens, full-bandwidth transformers "match or approach standard transformers trained with roughly 1.5× more tokens," at negligible per-token decoding overhead.
But the result I would actually build on is the quieter one. The feedback passes double as a training signal on the hidden states:
In later feedback passes, the top-layer state is shifted, fused into the input of subsequent positions, and can influence losses at multiple future positions through causal attention. Thus gradients from later predictions backpropagate into earlier hidden states, encouraging them to be reusable as inputs rather than merely predictive at the output layer.
In the ordinary objective, the top-layer state is supervised only through the next token. Here it is also supervised by whether it is useful to consume. And the payoff survives without the mechanism:
Empirically, this improves pre-training data efficiency even when latent feedback is not used at decoding time.
So there is a version of this that costs nothing at serving time: train with the feedback objective, decode normally, keep the representation gains. That is a much easier thing to adopt than a new decoding loop, and it is the finding most likely to show up in someone else's model.
The result that gets destroyed
On the base model, latent-feedback decoding produces markedly shorter reasoning traces at equal or better accuracy — exactly what the bandwidth argument predicts, since computation that would have to be spelled out can ride the hidden state instead.
Then:
Notably, the effect disappears after instruction tuning. We attribute this to the tuning data being off-policy with respect to latent-feedback decoding: the target traces were produced by (and imitate the verbosity of) standard token-by-token reasoning, so fitting them re-imposes the fully verbalized style regardless of what the state can carry.
This is the most interesting paragraph in the paper and it is reporting a failure.
A capability was trained in and then trained back out — by imitation data written by models that did not have it. The traces in every instruction-tuning set were produced under the old constraint, so they encode verbosity that the new architecture makes unnecessary, and fitting them teaches the model to keep paying a cost it no longer owes.
The fix they name is on-policy post-training under latent feedback, left to future work. The general shape of the problem is not specific to this paper: architectural capabilities can be erased by post-training data that predates them, and nobody notices, because the benchmark still passes.
What it does not establish
The two limitations are the authors' own, stated plainly.
Everything is at 1B parameters. Their intuition is that deeper models should benefit more, since a deeper stack's top-layer state carries more — but that is a hypothesis, and 1B is small enough that a 1.5× data-efficiency gain could plausibly shrink or grow at scale.
The feedback schedule is a heuristic. No ablation on how long the recurrence phase should run, no principled way to choose the number of recurrence steps; they point at Jacobi-iteration convergence diagnostics as a possible route.
I would add a third: the state-tracking probes that verify the extra bandwidth is used — completion tracking and delayed memory — are synthetic diagnostics built for the purpose. They show the channel carries something. They do not apportion the 1.5× between the wider channel and the extra training signal, and those are separable ideas with very different deployment costs.
Why the framing sticks
Almost a decade ago, Breaking the Softmax Bottleneck made a structurally identical argument about the other end of the model: the output layer factorizes through a matrix of rank at most the hidden size, so no matter how good your representations are, the distribution you can express is capacity-limited by a shape.
This paper makes the same kind of argument about the feedback path. Not "the model is not smart enough" but "the pipe is too narrow, and everything you have interpreted as a reasoning strategy is partly an adaptation to the pipe."
Whether or not latent feedback is the right fix, that is a productive way to look at a decoder. The interesting question it leaves open is how much of what we currently call reasoning is thinking, and how much is just a model talking to itself because that is the only channel it has.