LOTUS: reasoning in the hidden states, not the token stream

2026-07-15 · 10 min · llm · reasoning · chain-of-thought · latent-reasoning · inference-optimization · explainer

The way a reasoning model earns its answer is by writing out its work: an explicit chain of thought (CoT), one token at a time, before it commits to a final answer. That is where the latency goes. Each of those intermediate tokens is a full sequential decode step — a memory-bound pass over the growing KV cache — so the more the model thinks, the slower it answers.

Latent CoT is the tempting alternative: do the multi-step reasoning inside the model's hidden states, replacing decoded tokens with continuous representations, and skip the token-by-token bottleneck entirely. The problem is that it has never quite worked at scale. Methods like Coconut, CODI, and SIM-CoT match explicit CoT on small models, but beyond 1B parameters no latent method keeps up on math reasoning, and the gap widens as the backbone grows. LOTUS — Looped Transformers with parallel supervision on latents, from Ying Fan, Anej Svete, and Kangwook Lee — is, to the authors' knowledge, the first latent-CoT method to close that gap at the 3B scale, while cutting the thought phase by 2.5×–6.9×.

It gets there by fixing the two things the authors argue were holding latent CoT back.

(P1) Sequential generation. Coconut, CODI, and SIM-CoT still produce their latent tokens autoregressively — the sequential bottleneck is still there, just moved into latent space.
(P2) No CoT grounding. Without supervision that aligns each latent position to a specific gold reasoning step, the latent trace drifts and destabilizes as the model gets bigger.

The loop: one weight set, R passes, K blocks in parallel

LOTUS builds a padded latent region into the prompt. Between two learnable delimiters ⟨BoT⟩ and ⟨EoT⟩ it inserts $K$ blocks of $c$ shared, learnable ⟨lat⟩ tokens — a fixed $K\cdot c$ latent positions (the deployed config is $K=6$ , $c=25$ , so 150 positions). The question $Q$ sits before ⟨BoT⟩; the answer $A$ comes after ⟨EoT⟩.

The reasoning then happens by looping the base language model over that region. Let $E$ be the learnable latent embeddings and $f_\theta$ the ordinary LM backbone. Starting from the latents, LOTUS reuses the same weights for $R$ iterations, adding the previous pass's output back in each time:

h^{(0)} = f_\theta\!\big(E \mid C_{\text{pre}}\big), \qquad h^{(t)} = f_\theta\!\big(E + h^{(t-1)} \mid C_{\text{pre}}\big), \quad t = 1,\dots,R

where $C_{\text{pre}}$ is the reused KV cache of the question. This is a recurrent-depth (looped) Transformer: it adds computation depth by reusing parameters, not by adding them. The crucial property is that all $K\cdot c$ latent positions are refined together on each pass — so the whole thought phase is $R$ sequential forward passes, not one pass per generated token. Scrub the loop and watch the latents sharpen, then read out at the final iteration:

looped padded transformer · one thought phaseillustrative

iteration t = 3 of R = 6

forward passes so far: 3 · positions per pass: 6×25 = 150

loop iteration (drag to t = R for readout)

The reasoning trace is not decoded token-by-token. All 6×25 = 150 latent positions are refined together on every pass, and the whole thought phase is just R = 6 loops of the same weights. At the final iteration the base LM head reads each position out to its gold CoT-step token — that readout is exactly what the step loss supervises.

That parallelism is the answer to (P1). Where an autoregressive latent method spends a forward pass per latent token — the same shape of bottleneck that makes multi-token prediction and diffusion language models attractive — LOTUS spends only $R$ passes for the entire trace, regardless of how many tokens that trace would have been.

The other way to see this is as a network. A looped Transformer is a recurrent-depth network: roll it up and it is one block with a loop-back edge; unroll it and it is an effective $R$ -deep stack of the same weights, with the latents $E$ fed back in at every pass. The depth is real — each pass is a full forward through the backbone — but the parameter count never grows past $1\times$ . Toggle between the rolled and unrolled views, and drag the unroll depth:

looped transformer · depth from weight reusenetwork view

view

effective depth 6× layers · parameters 1× (shared)

unroll depth (drag to t = R)

Rolled up, LOTUS is one block with a loop-back edge. Unrolled, it is an effective R-deep network of the same weights — depth without new parameters — with the input latents E fed back into every pass (the E + h⁽ᵗ⁻¹⁾ residual, dashed). That is what lets a 3B backbone reason at a depth its parameter count alone would not buy.

That is the whole trick behind "add computation depth by reusing parameters, not by adding them": a 3B backbone reasons at a depth its parameter budget alone would not buy, because depth here is $R$ passes through shared weights rather than $R$ times the weights.

Parallel supervision: grounding each latent in its gold step

The loop alone is not enough; the latents need to be told what to compute. This is the answer to (P2), and it is the part that makes LOTUS more than "a looped model." After the final iteration, LOTUS reads each post-loop latent position $h^{(R)}_{i,j}$ through the base LM head $f_{\text{head}}$ and trains it, with cross-entropy, toward the gold CoT-step token that belongs in that slot. Each gold CoT step $i$ is tokenized and padded/truncated to $c$ tokens $T_{i,\cdot}$ , and every position is supervised at once:

\mathcal{L}_{\text{step}} = \frac{1}{N_{\text{step}}}\sum_{i=1}^{K}\sum_{j=1}^{c} \operatorname{CE}\!\big(f_{\text{head}}(h^{(R)}_{i,j}),\, T_{i,j}\big)

This is direct, position-aligned supervision to real reasoning tokens — much like ordinary explicit-CoT supervision — rather than the indirect hidden-state or KV-cache distillation used by earlier parallel-latent methods (PCCoT, KaVa). A separate final forward pass then supervises the answer against the latents:

\mathcal{L} = \mathcal{L}_{\text{ans}} + \lambda_{\text{step}}\,\mathcal{L}_{\text{step}}

LOTUS architecture, two panels. (a) Looped forward: the input row is Q, BoT, a run of lat tokens grouped into block 1 through block K, then EoT. A single base LM f_theta box is applied with a curved loop arrow labelled ×R, producing post-loop hidden states h(R), one per latent position, which are read through f_head up to a row of gold CoT token boxes. (b) Final forward: the post-loop latents are inserted back into the sequence and one more pass through f_theta produces answer hidden states z, read through f_head to answer tokens A1, A2, supervised by L_ans. — LOTUS refines K blocks of latent tokens in parallel over R looped passes of one shared base LM, then reads each latent position out through the base LM head to its gold CoT-step token (L_step). A final forward pass produces the answer (L_ans) (Fan et al., 2026, Figure 2).

The paper frames why both losses are needed with a Parallel Chain Likelihood (PCL) view. Because the step loss factorizes over positions rather than autoregressively, it induces

p_\theta^{\text{PCL}}(T\mid Q) = \prod_{i=1}^{K}\prod_{j=1}^{c} p_\theta(T_{i,j}\mid Q)

The two losses then split the work: $\mathcal{L}_{\text{step}}$ provides coverage — it puts probability mass on the correct gold token at each position — while $\mathcal{L}_{\text{ans}}$ provides selection — it forces the jointly-computed latents to actually support the right answer. The ablation makes the split concrete: with only $\mathcal{L}_{\text{step}}$ , the model's post-loop latents recover the gold top-1 token just 9.1% of the time; with only $\mathcal{L}_{\text{ans}}$ , 9.4%; with both, 70.9% (NLL 3.07 vs 9.29 and 5.97). Neither loss alone builds a readable, correct latent trace.

Why it is fast

The efficiency story is simple once the loop is clear. Explicit CoT's thought-phase latency scales with how much it writes; LOTUS's scales with $R$ , which is fixed. So the win grows exactly when the rationale gets verbose. On Llama-3.2-3B the paper measures the thought phase directly — drag the playhead and watch LOTUS finish while explicit CoT is still decoding:

thought-phase latency · Llama-3.2-3B, ms/examplepaper, Table 2 & 3

rationale

thought speedup 2.5× · at 210 ms: LOTUS done, explicit 62%

wall-clock (drag)

Explicit CoT's thought phase grows with how much it writes — swap the compact math chain for a verbose natural-language rationale and it balloons from 338.8 to 963.6 ms. LOTUS barely moves (133.0 → 140.8 ms): the trace is always R = 6 parallel passes, so the same mechanism buys a 2.5× speedup on math and a 6.9× speedup on natural language. (Query-prefill and answer phases are near-identical across methods; only the thought phase moves.)

On compact math-expression CoT the thought phase drops from 338.8 ms to 133.0 ms (2.5×), and total latency from 384.2 ms to 181.2 ms (about 2.1× overall). Swap in verbose natural-language rationales and explicit CoT balloons to 963.6 ms while LOTUS barely moves to 140.8 ms — a 6.9× thought-phase speedup, at essentially the same accuracy (68.13% vs 68.41%). The query-prefill and answer phases are nearly identical across methods; only the thought phase moves.

The numbers

Held against explicit CoT and the strongest latent baselines on GSM8K (Llama-3.2-3B, in-domain), LOTUS lands within about a point and a half of explicit CoT and clearly ahead of the latent baselines:

GSM8K accuracy, Llama-3.2-3B in-domain (%)

Explicit CoT

71.5%

LOTUS

70%

LOTUS + CODI

70.6%

CODI + SIM-CoT

62.3%

020406080

The pattern holds across backbones — GPT-2 (LOTUS 44.1 vs explicit 42.7), Llama-1B (57.3 vs 58.4), Llama-3B (70.0 vs 71.5) — so unlike prior latent methods the gap does not widen with scale. And on the out-of-domain average (GSM-Hard, MultiArith, SVAMP), LOTUS actually edges ahead of explicit CoT, 63.9 vs 62.1, led by a near-perfect 99.9% on MultiArith and 75.7% on SVAMP.

Thought-phase speedup vs explicit CoT, Llama-3B (×)

math CoT

2.5×

natural-language CoT

6.9×

total (math)

2.1×

02468

How deep does the loop need to be?

Loop depth $R$ is LOTUS's compute knob, and reasoning turns out to need a real amount of it. Train the model at increasing $R$ and accuracy climbs steeply — a shallow loop simply cannot fit multi-step arithmetic:

accuracy vs loop depth R · Llama-3B, GSM8Kpaper, Table 5 & 6

curve

each point is a separately trained model

loop depth R (drag)

Depth is the knob. Trained too shallow, the loop simply cannot fit multi-step arithmetic — R = 2 lands at 14.6%. Give it R = 6 and accuracy reaches 70.0%. But depth is not free test-time compute: take the R = 6 model and run it deeper and accuracy dips (R = 7 → 69.3%) — LOTUS reasons best at the depth it was trained for, not beyond.

Two honest wrinkles live in this chart. First, depth is not free test-time compute you can dial up after training: take the model trained at $R=6$ and run it at $R=7$ and accuracy dips to 69.3% — LOTUS reasons best at the depth it was trained for. Second, the parallel width $c$ is nearly free where depth is not: sweeping $c$ from 1 to 50 tokens per block (a 50× change in latent positions) moves the thought phase by only about 30 ms — 110.9 ms to 141.2 ms — because those positions are processed in parallel. A single token per step (c=1) is too narrow (51.4%), but moderate widths (c=25–30) saturate at 70%.

Does it actually reason, or memorize?

Because LOTUS reads its latents through the ordinary LM head, you can literally decode the thought — the same trick behind interpretability tools that unembed intermediate activations. The post-loop latents recover the gold CoT at 70.9% top-1 / 85.8% top-5. More telling is a multi-path test: for a question with a trained gold chain (G) and an unseen-but-valid alternative chain (U), the readout orders their likelihoods $G \ll U \ll \text{random}$ (NLL 0.07, 4.28, 8.16), assigning graded probability to valid-but-never-seen reasoning. That ordering is the paper's evidence that the latents encode reasoning structure, not just a memorized string.

warning

Read the setup before believing the headline. (1) Scope is narrow: every result is math word problems (GSM8K-family, trained on GSM8k-Aug); the authors explicitly flag transfer to other domains as open. (2) The budget is fixed: $K$ , $c$ , and $R$ are hyperparameters set to cover the expected step count — chains longer than $K$ steps fall back to autoregressive completion, and making the budget adaptive is listed as future work, not a solved problem. (3) "Bridges the gap" means parity, not a win: LOTUS is ~1.5 points behind explicit CoT in-domain at 3B (70.0 vs 71.5); it needs the LOTUS+CODI combo to get within a point, and it leads only on the OOD average. (4) Speedups are the paper's own measurements on Llama-3B (H-class GPU), and the flattering 6.9× is specifically the verbose natural-language regime; the compact-math number is 2.5×. (5) Baselines are author-selected latent methods (Coconut, CODI, SIM-CoT, PCCoT, KaVa) and one explicit-CoT reference — not a broad frontier-model comparison.

The take

LOTUS's real contribution is a clean recombination: take a looped padded Transformer (depth from weight reuse, all latent positions refined in parallel) and give it direct, position-aligned supervision to gold CoT tokens through the base LM head. The loop kills the sequential bottleneck that every prior latent method kept; the supervision kills the drift that made latent reasoning fall apart at scale. Together they do something no earlier latent-CoT method managed — stay on the explicit-CoT accuracy curve at 3B — while turning a variable, write-everything thought phase into a fixed $R$ -pass one.

The honest frame is that this is a parity-with-a-speedup result on math, bounded by a thinking budget you have to choose in advance. Whether a fixed $K\cdot c\cdot R$ box holds up when problems demand more steps than you budgeted — and whether the story survives outside arithmetic — are the open questions the authors name themselves. But as a demonstration that latent reasoning can finally keep pace with explicit reasoning at scale, and answer 2.5–6.9× faster while doing it, LOTUS is the first latent-CoT method that clears the bar.

Built on Bridging the Gap Between Latent and Explicit Reasoning with Looped Transformers (Fan, Svete, Lee; 2026). All accuracy, latency, and ablation figures are quoted from the paper (Llama-3.2-3B unless noted; GSM8K-family math benchmarks; deployed config $K=6$ , $c=25$ , $R=6$ ). The interactive diagrams are illustrations of the mechanism; the gold-CoT tokens in the loop diagram are illustrative.