Towards looped models done right: what actually separates Ouro from Huginn

Ask why Huginn-style looped transformers tend to beat Ouro-style ones and most people reach for the same answer: random state initialization. It is the assumption inherited wholesale from deep equilibrium models — randomize the recurrent state so the loop cannot just memorize a fixed point, and the model is forced to learn a genuinely path-independent computation. It sounds right. It is also, according to a new controlled-ablation report from the Institute of Foundation Models (IFM), mostly wrong.

Towards Looped Models Done Right does not propose a new looped architecture. It audits the two existing lineages this site has already covered — the padded-latent loop in LOTUS and the shipped two-pass loop in Nanbeige4.2-3B both descend from this same family of looped / recurrent-depth transformers — and asks a narrower, more useful question: Ouro and Huginn differ along three design axes at once, so which one is actually doing the work? The answer, walked through below, is not the one folk wisdom would guess.

One formalism, two lineages, three tangled axes

Both Ouro and Huginn are instances of the same tied-iterative model. Take token embeddings, run them through a prelude, loop a shared recurrent body $T$ times, then run a coda:

$P_\theta$ is the prelude, $R_\theta$ the tied recurrent body (the only part that repeats), $C_\theta$ the coda, $\phi_\theta$ the state initializer (optionally seeded with noise $\boldsymbol\xi$), and $W_\theta$ the per-step write that decides how much of $\mathbf e$ gets re-injected at each pass.

Set every one of $P_\theta$, $W_\theta$, $\phi_\theta$ to the identity and you get Ouro-style: the entire network is the recurrent body, looped over the full sequence, with $\mathbf z_0 = \mathbf e$ directly. Untie a real prelude and coda from the loop, persistently re-inject $\mathbf e$ into the core at every pass, and optionally randomize the initial state, and you get Huginn-style. Three independent knobs — iteration envelope, input interface, latent-state design — get flipped between the two architectures simultaneously in the existing literature. Nobody had isolated which flip mattered.

expand

The controls that make this an ablation, not a vibe check

A myth-busting result is only as good as what it holds constant. IFM's models are matched on logical depth — Ouro-style loops a 28-block stack four times, Huginn-style runs an 8-block prelude, a 12-block core eight times, and an 8-block coda; both total 112 block executions — and on parameters (730M stored / 2.9B unrolled-equivalent dense; 8B-resident / 0.8B-active, 32B/3.2B unrolled-equivalent for the MoE runs), tokens (TxT360, swept across 58/115/230/460 tokens-per-parameter), and a ten-benchmark suite spanning knowledge (ARC-C, HellaSwag, MMLU, TriviaQA), context reasoning (BBH-CoT, DROP), math (GSM8K, MATH500), and code (HumanEval+, MBPP+). Whichever topology wins a given comparison, it isn't winning on a hidden depth or size advantage:

With the controls fixed, IFM walks the transformation path from Ouro to Huginn one axis at a time — sandwich envelope, then input injection, then random state init — and measures what each addition actually buys.

Q1: does untying the prelude and coda matter?

Yes, and only for a specific kind of task. Adding a sandwich envelope — untying a prelude and coda from the loop so only the middle core repeats — lifts MATH500 by 12.00 points and DROP by 2.61 points at the 460 tokens-per-parameter budget, and the gain persists across all four token budgets tested. But knowledge-heavy benchmarks and strict-output-contract tasks like code show no consistent gain, sometimes a small decline. Read plainly: the envelope helps instance-conditioned, multi-step reasoning. It does not help stored-knowledge recall or spec-compliant code generation, and the report is upfront that it shouldn't be expected to.

Q2: does persistent input injection matter?

Also yes, and it's the widest-reaching single change in the whole report. Writing the prelude's representation $\mathbf e$ into the core at every pass — not just once at the start — uses a learned, per-channel gate:

On the middle-loop (sandwich) topology, adding this write lifts MMLU +2.53, BBH-CoT +6.63, DROP +1.39, HumanEval+ +5.49, MBPP+ +4.23 — five benchmarks, all up, some by a lot. Bolt the same write onto the full-stack Ouro topology (injecting raw token embeddings instead of a prelude-encoded $\mathbf e$) and the same-direction gains show up smaller (MMLU +1.79, BBH-CoT +1.80, DROP +0.91, HumanEval+ +2.44, MBPP+ +4.50) — evidence the effect is real and not an artifact of one topology.

It is not a free lunch. The same write hurts quantitative reasoning: middle-loop MATH500 drops 3.60 points, GSM8K drops 2.51 (the full-stack Ouro version is less damaged: MATH500 −1.60, GSM8K +0.53). Persistently re-showing the model its own input, it turns out, competes with letting the loop's state evolve freely enough to carry a multi-step derivation.

expand

Put the envelope and the injection together and the combined model beats full-stack Ouro on 8 of the 10 benchmarks (losing only ARC-Challenge and HellaSwag). Step through both changes yourself — the diagram below is my own redrawing of the same construction path, with each stage's measured delta attached so you can see exactly which wire produced which number, before the third, more surprising change gets added:

Q3: the myth — random state init and shared H/L hierarchies

This is the report's contrarian core. Swap the direct initial state $\mathbf z_0 = \mathbf e$ for a randomly sampled one, $\mathbf z_0 \sim \mathcal N(0, I/d)$ — the equilibrium-model-inherited move everyone assumes is load-bearing — and two benchmarks improve (ARC-C +3.34, GSM8K +1.22) while four get worse by more than a point (MMLU, MATH500, HumanEval+, MBPP+). Net: direct init wins 6 of 10 benchmarks, and it's cheaper, since it skips sampling noise at every forward pass. The report's own words: "random initialization is not a necessary ingredient for loop language models... [it] should instead be viewed as a task- and objective-dependent inductive bias, rather than as a universally beneficial design choice."

A second candidate "obviously helps" ingredient fares no better. HRM/TRM-style hierarchies split the loop into a slow high-level state and a fast low-level state cycling underneath it; IFM tests a version that shares one recurrent body across both states (isolating the state-hierarchy idea from the separate-modules idea) and finds gains over a point on three benchmarks, losses over a point on three more — MATH500 hit hardest — and roughly flat on the rest. Their conclusion: "a shared-module H/L hierarchy provides no consistent benefit." (They flag, honestly, that this doesn't rule out a separately-parameterized HRM-Text-style version — that variant is "still under evaluation.")

Here is the believed-important story against what got measured, for both:

The net effect: two wires did almost all the work

Chain every change together — Ouro, plus envelope, plus injection, plus random init — and you land on full Huginn, which does beat Ouro on all ten dense benchmarks at the 730M/336B-token setting. But laid out per-benchmark across the whole construction path, the shape of the win is obvious: most of the climb happens in the first two steps, and the last step (random init) barely moves several benchmarks and actively costs a few.

expand

Does the story change at MoE scale?

The two levers that mattered — envelope and injection — hold up when the recurrent body becomes a mixture-of-experts. At 8B-resident / 793.9M-active parameters (500B tokens, top-2 routing, 25 experts), Huginn-MoE beats Ouro-MoE on 8 of 10 benchmarks, with the largest gains on GSM8K (+4.70) and MATH500 (+3.60). It also routes more evenly — a normalized load-balancing loss where lower means more balanced:

A causal check backs up that the routing difference is meaningful, not noise: force loop iterations 2 through 8 to reuse iteration 1's expert identities (keeping the iteration-specific mixture weights) and accuracy drops on all six evaluated tasks. Whatever the loop is learning to route each pass, it matters.

Against a 112-layer feedforward MoE reference (32B resident parameters), the feedforward model still wins overall — 7 of 10 benchmarks — but Huginn-MoE beats it on DROP and GSM8K and matches it on MATH500, while using 75% fewer resident parameters. The mean gap to the feedforward reference shrinks from 4.96 points in the dense setting to 1.71 points in the MoE setting. Looping doesn't close the gap to a much bigger feedforward model outright, but MoE narrows it substantially — the same shape of result as Nanbeige's own parameter-efficiency argument, below.

Set against a shipped model: Nanbeige4.2-3B

The most useful check on any ablation study is an independent result that wasn't trying to test the same hypothesis. Nanbeige4.2-3B's technical report is exactly that: a production model that made its own looping decisions under deployment pressure, not a controlled academic sweep.

Nanbeige's architecture is closer to Ouro-style — a homogeneous loop over the full stack, run twice — and its report reached three conclusions of its own: two passes is the sweet spot (more loop count bought little and made training less stable), training the looped architecture from scratch beats upcycling a pretrained feedforward model into one, and sharing the KV cache across passes underperformed, so they paid full attention cost at every pass rather than take the cheaper shortcut.

None of Nanbeige's three findings directly tests IFM's three axes — Nanbeige never tried an untied prelude/coda, persistent injection, or random state init — so this isn't a replication in either direction. But the two reports rhyme in an interesting way: every place Nanbeige tested a cheap shortcut inside the loop (share the KV cache, upcycle instead of retraining, add more passes without changing the topology), the shortcut lost. Every place IFM tested a richer per-pass mechanism (untie the envelope, inject persistently), it won — and the one change that added complexity without adding real per-pass information (random init) was the one that didn't clearly help. Read together, the two reports point at the same underlying rule: what a looped model does each pass — how much fresh computation and fresh input it gets — matters more than how many times it loops or how its state gets seeded. Where they don't overlap at all is loop count itself: Nanbeige's "two is enough" is a statement about a plain full-stack loop; IFM's Ouro-style baseline already runs four passes and Huginn-style runs eight core iterations inside a smaller envelope, so the two reports are sweeping different variables and shouldn't be read as agreeing or disagreeing on "how many loops."

There's a second, more mechanical echo. LOTUS — the site's other looped-transformer piece — already does something IFM's ablation independently flags as one of the two levers that matter: every LOTUS iteration recomputes $\mathbf e + \mathbf h^{(t-1)}$, persistently feeding the fixed input embeddings back into the loop rather than only conditioning on them once. That's a specific instance of the same principle behind IFM's write operator $D(\mathbf z_t, \mathbf e)$: keep re-showing the loop its input. LOTUS applies that idea inside a frozen-backbone latent-reasoning setup at inference time rather than IFM's from-scratch pretraining setup, so the two aren't directly comparable — but it's a second, independent place persistent input injection shows up as doing real work.

What to trust, and what to hold loosely

The take

The intuitive story about looped transformers has always centered on the recurrent state itself — randomize it, and the model is forced to learn something more general. IFM's controlled ablations say that story is backwards for at least these two designs at this scale: the state's initialization is close to a wash, sometimes a net loss, and a fashionable H/L hierarchy adds nothing consistent once you control for everything else. What actually separates a strong looped model from a weak one is much less exotic — untie a prelude and coda from the loop so the recurrent core can specialize, and keep showing that core its input at every pass instead of just once. Neither idea needs a random number generator.

That's a genuinely useful result for anyone building a looped model today, and it lines up with what Nanbeige found the hard way in production: cheap shortcuts inside the loop (shared KV cache, more passes without restructuring, upcycling instead of retraining) tend to cost you, while spending real compute and real information on each pass tends to pay. The caveat that matters most is the one IFM states themselves — this is one group's ablation, at sub-billion-to-8B scale, with no code out yet and no arXiv paper. "Part I" means there's more coming. Whether the random-init myth holds at 100B+ parameters, and whether a properly separate-module HRM-Text hierarchy fares better than the shared one tested here, are open questions the authors name as future work, not settled ones.

Source: Towards Looped Models Done Right — Part I: Topology, Input Injection, Recurrent-State Design (Huang, Shi, Chen, Wen, Liu, Xing, Ma; Institute of Foundation Models, 2026). All benchmark deltas, equations, and figures are quoted or reproduced from the report; the matched-depth, construction-path, and believed-vs-measured diagrams are my own illustrations of the same numbers.