# Towards looped models done right: what actually separates Ouro from Huginn

> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/looped-models-done-right
> date: 2026-08-03
> tags: llm, looped-transformers, recurrent-depth, ablation-study, architecture, mixture-of-experts, explainer
Ask why Huginn-style looped transformers tend to beat Ouro-style ones and most people reach for the
same answer: random state initialization. It is the assumption inherited wholesale from deep
equilibrium models — randomize the recurrent state so the loop cannot just memorize a fixed point,
and the model is forced to learn a genuinely path-independent computation. It sounds right. It is also,
according to a new controlled-ablation report from the Institute of Foundation Models (IFM), mostly
wrong.

**Towards Looped Models Done Right** does not propose a new looped architecture. It audits the two
existing lineages this site has already covered — the padded-latent loop in
[LOTUS](/articles/lotus-latent-reasoning) and the shipped two-pass loop in
[Nanbeige4.2-3B](/articles/nanbeige-4-2-3b) both descend from this same family of
[looped / recurrent-depth transformers](/architectures) — and asks a narrower, more useful question:
Ouro and Huginn differ along *three* design axes at once, so which one is actually doing the work? The
answer, walked through below, is not the one folk wisdom would guess.

<Callout type="warn">
**Read this as a first look, not a finished paper.** This report lives only as a Notion "living blog,"
explicitly billed as Part I of a series and continuously updated — there is no arXiv listing. Code is
marked "Release Soon," meaning nobody outside IFM can rerun these numbers yet, and there has been no
third-party replication. Every result below comes from ablations run by one group, at 730M dense /
8B-resident MoE scale — a real, carefully controlled experiment, but not yet an independently checked
one, and not yet evidence about frontier scale.
</Callout>

## One formalism, two lineages, three tangled axes

Both Ouro and Huginn are instances of the same tied-iterative model. Take token embeddings, run them
through a prelude, loop a shared recurrent body $T$ times, then run a coda:

$$
\mathbf e = P_\theta(\mathbf x_0),\quad
\mathbf z_0 = \phi_\theta(\mathbf e, \boldsymbol\xi),\quad
\tilde{\mathbf z}_t = W_\theta(\mathbf z_t, \mathbf e),\quad
\mathbf z_{t+1} = R_\theta(\tilde{\mathbf z}_t),\quad
\mathbf h = C_\theta(\mathbf z_T)
$$

$P_\theta$ is the prelude, $R_\theta$ the tied recurrent body (the only part that repeats), $C_\theta$
the coda, $\phi_\theta$ the state initializer (optionally seeded with noise $\boldsymbol\xi$), and
$W_\theta$ the per-step write that decides how much of $\mathbf e$ gets re-injected at each pass.

Set every one of $P_\theta$, $W_\theta$, $\phi_\theta$ to the identity and you get **Ouro-style**: the
entire network is the recurrent body, looped over the full sequence, with $\mathbf z_0 = \mathbf e$
directly. Untie a real prelude and coda from the loop, persistently re-inject $\mathbf e$ into the core
at every pass, and optionally randomize the initial state, and you get **Huginn-style**. Three
independent knobs — iteration envelope, input interface, latent-state design — get flipped between the
two architectures simultaneously in the existing literature. Nobody had isolated which flip mattered.

<Figure
  src="/articles/looped-models-done-right/fig1.png"
  alt="Three architecture diagrams side by side. (a) Feedforward: token embedding through a stack of distinct blocks B1 through BL to output. (b) Ouro-style: token embedding into a single tied block R-theta looped T times, straight to output. (c) Huginn-style: token embedding through prelude P-theta to representation e, which feeds a write operator W-theta, an initializer producing z0 from the data distribution, a tied loop R-theta applied T times with e fed back in via W-theta at every pass, then a coda C-theta to output."
  caption="IFM Research's own diagram of the three topologies this ablation compares: a plain feedforward stack, the homogeneous Ouro-style loop, and the Huginn-style prelude–loop–coda envelope with its explicit write operator (IFM Research, 2026)."
/>

## The controls that make this an ablation, not a vibe check

A myth-busting result is only as good as what it holds constant. IFM's models are matched on logical
depth — Ouro-style loops a 28-block stack four times, Huginn-style runs an 8-block prelude, a 12-block
core eight times, and an 8-block coda; both total 112 block executions — and on parameters (730M
stored / 2.9B unrolled-equivalent dense; 8B-resident / 0.8B-active, 32B/3.2B unrolled-equivalent for the
MoE runs), tokens (TxT360, swept across 58/115/230/460 tokens-per-parameter), and a ten-benchmark suite
spanning knowledge (ARC-C, HellaSwag, MMLU, TriviaQA), context reasoning (BBH-CoT, DROP), math (GSM8K,
MATH500), and code (HumanEval+, MBPP+). Whichever topology wins a given comparison, it isn't winning on
a hidden depth or size advantage:

<MatchedDepth />

With the controls fixed, IFM walks the transformation path from Ouro to Huginn one axis at a time —
sandwich envelope, then input injection, then random state init — and measures what each addition
actually buys.

## Q1: does untying the prelude and coda matter?

Yes, and only for a specific kind of task. Adding a sandwich envelope — untying a prelude and coda from
the loop so only the middle core repeats — lifts **MATH500 by 12.00 points** and **DROP by 2.61 points**
at the 460 tokens-per-parameter budget, and the gain persists across all four token budgets tested. But
knowledge-heavy benchmarks and strict-output-contract tasks like code show no consistent gain, sometimes
a small decline. Read plainly: the envelope helps *instance-conditioned, multi-step reasoning*. It does
not help stored-knowledge recall or spec-compliant code generation, and the report is upfront that it
shouldn't be expected to.

## Q2: does persistent input injection matter?

Also yes, and it's the widest-reaching single change in the whole report. Writing the prelude's
representation $\mathbf e$ into the core at every pass — not just once at the start — uses a learned,
per-channel gate:

$$
D(\mathbf z_t, \mathbf v) = \boldsymbol\alpha \odot \mathbf z_t + \boldsymbol\delta \odot \mathbf W_{\text{in}}\mathbf v,
\qquad \boldsymbol\delta = \operatorname{softplus}(\mathbf b_\delta),
\qquad \boldsymbol\alpha = \exp\{-\boldsymbol\delta \odot \exp(\mathbf a)\}
$$

On the middle-loop (sandwich) topology, adding this write lifts **MMLU +2.53, BBH-CoT +6.63, DROP +1.39,
HumanEval+ +5.49, MBPP+ +4.23** — five benchmarks, all up, some by a lot. Bolt the same write onto the
full-stack Ouro topology (injecting raw token embeddings instead of a prelude-encoded $\mathbf e$) and
the same-direction gains show up smaller (MMLU +1.79, BBH-CoT +1.80, DROP +0.91, HumanEval+ +2.44,
MBPP+ +4.50) — evidence the effect is real and not an artifact of one topology.

It is not a free lunch. The same write **hurts** quantitative reasoning: middle-loop MATH500 drops
**3.60 points**, GSM8K drops **2.51** (the full-stack Ouro version is less damaged: MATH500 −1.60,
GSM8K +0.53). Persistently re-showing the model its own input, it turns out, competes with letting the
loop's state evolve freely enough to carry a multi-step derivation.

<Figure
  src="/articles/looped-models-done-right/fig2.png"
  alt="Three architecture panels showing a construction path with arrows between them. (a) Full-stack loop: token embedding directly into a tied R-theta block looped 4 times to output. (b) Middle loop (sandwich): token embedding through P-theta, then a smaller tied R-theta looped 8 times, then C-theta, to output. (c) Middle loop plus input injection: same as (b) but with an explicit e node and W-theta write operator feeding the tied loop at every one of the 8 passes, with z0 equal to e."
  caption="IFM Research's own construction-path diagram: full-stack Ouro, then the sandwich envelope alone, then envelope plus persistent input injection — the same two steps walked interactively below (IFM Research, 2026)."
/>

Put the envelope and the injection together and the combined model beats full-stack Ouro on **8 of the
10 benchmarks** (losing only ARC-Challenge and HellaSwag). Step through both changes yourself — the
diagram below is my own redrawing of the same construction path, with each stage's measured delta
attached so you can see exactly which wire produced which number, before the third, more surprising
change gets added:

<EnvelopeStepper />

## Q3: the myth — random state init and shared H/L hierarchies

This is the report's contrarian core. Swap the direct initial state $\mathbf z_0 = \mathbf e$ for a
randomly sampled one, $\mathbf z_0 \sim \mathcal N(0, I/d)$ — the equilibrium-model-inherited move
everyone assumes is load-bearing — and two benchmarks improve (**ARC-C +3.34, GSM8K +1.22**) while four
get worse by more than a point (**MMLU, MATH500, HumanEval+, MBPP+**). Net: **direct init wins 6 of 10
benchmarks**, and it's cheaper, since it skips sampling noise at every forward pass. The report's own
words: "random initialization is not a necessary ingredient for loop language models... [it] should
instead be viewed as a task- and objective-dependent inductive bias, rather than as a universally
beneficial design choice."

A second candidate "obviously helps" ingredient fares no better. HRM/TRM-style hierarchies split the
loop into a slow high-level state and a fast low-level state cycling underneath it; IFM tests a version
that **shares one recurrent body** across both states (isolating the state-hierarchy idea from the
separate-modules idea) and finds gains over a point on three benchmarks, losses over a point on three
more — MATH500 hit hardest — and roughly flat on the rest. Their conclusion: "a shared-module H/L
hierarchy provides no consistent benefit." (They flag, honestly, that this doesn't rule out a
separately-parameterized HRM-Text-style version — that variant is "still under evaluation.")

Here is the believed-important story against what got measured, for both:

<MythVsMeasured />

## The net effect: two wires did almost all the work

Chain every change together — Ouro, plus envelope, plus injection, plus random init — and you land on
full Huginn, which does beat Ouro on all ten dense benchmarks at the 730M/336B-token setting. But laid
out per-benchmark across the whole construction path, the shape of the win is obvious: most of the
climb happens in the first two steps, and the last step (random init) barely moves several benchmarks
and actively costs a few.

<Figure
  src="/articles/looped-models-done-right/fig3.png"
  alt="A ten-panel grid of line charts, one per benchmark (ARC-C, HSwag, MMLU, BBH-CoT, TQA, DROP, MATH500, GSM8K, HEval+, MBPP+), each plotting raw benchmark score against four construction stages labeled O, +P/C, +W, +z, with reference dashed lines for a 28-layer and a 112-layer feedforward baseline. Most benchmarks rise steeply from O to +P/C and +W, then plateau or dip slightly at +z."
  caption="IFM Research's raw benchmark scores across the same four-stage construction path — O (Ouro) through +P/C (envelope), +W (injection), +z (random init) — against 28-layer and 112-layer feedforward references. The plateau (and occasional dip) at the last step is the random-init myth, in the paper's own data (IFM Research, 2026)."
/>

## Does the story change at MoE scale?

The two levers that mattered — envelope and injection — hold up when the recurrent body becomes a
mixture-of-experts. At 8B-resident / 793.9M-active parameters (500B tokens, top-2 routing, 25 experts),
Huginn-MoE beats Ouro-MoE on 8 of 10 benchmarks, with the largest gains on **GSM8K (+4.70)** and
**MATH500 (+3.60)**. It also routes more evenly — a normalized load-balancing loss where lower means more
balanced:

<BenchBars
  title="MoE load-balancing loss at 500B tokens (lower = more balanced)"
  unit=""
  bars={[
    { label: "Ouro-MoE", value: 1.899 },
    { label: "112-layer feedforward MoE", value: 1.652 },
    { label: "Huginn-MoE", value: 1.571, highlight: true },
  ]}
/>

A causal check backs up that the routing difference is meaningful, not noise: force loop iterations 2
through 8 to reuse iteration 1's expert *identities* (keeping the iteration-specific mixture weights)
and accuracy drops on all six evaluated tasks. Whatever the loop is learning to route each pass, it
matters.

Against a **112-layer feedforward MoE** reference (32B resident parameters), the feedforward model still
wins overall — 7 of 10 benchmarks — but Huginn-MoE beats it on DROP and GSM8K and matches it on MATH500,
while using **75% fewer resident parameters**. The mean gap to the feedforward reference shrinks from
4.96 points in the dense setting to 1.71 points in the MoE setting. Looping doesn't close the gap to a
much bigger feedforward model outright, but MoE narrows it substantially — the same shape of result as
Nanbeige's own parameter-efficiency argument, below.

## Set against a shipped model: Nanbeige4.2-3B

The most useful check on any ablation study is an independent result that wasn't trying to test the
same hypothesis. [Nanbeige4.2-3B's technical report](/articles/nanbeige-4-2-3b) is exactly that: a
production model that made its own looping decisions under deployment pressure, not a controlled
academic sweep.

Nanbeige's architecture is closer to Ouro-style — a homogeneous loop over the full stack, run twice —
and its report reached three conclusions of its own: **two passes is the sweet spot** (more loop count
bought little and made training less stable), **training the looped architecture from scratch beats
upcycling** a pretrained feedforward model into one, and **sharing the KV cache across passes
underperformed**, so they paid full attention cost at every pass rather than take the cheaper shortcut.

None of Nanbeige's three findings directly tests IFM's three axes — Nanbeige never tried an untied
prelude/coda, persistent injection, or random state init — so this isn't a replication in either
direction. But the two reports rhyme in an interesting way: every place Nanbeige tested a cheap shortcut
inside the loop (share the KV cache, upcycle instead of retraining, add more passes without changing the
topology), the shortcut lost. Every place IFM tested a richer per-pass mechanism (untie the envelope,
inject persistently), it won — and the one change that added complexity without adding real per-pass
information (random init) was the one that didn't clearly help. Read together, the two reports point at
the same underlying rule: what a looped model does *each pass* — how much fresh computation and fresh
input it gets — matters more than how many times it loops or how its state gets seeded. Where they don't
overlap at all is loop count itself: Nanbeige's "two is enough" is a statement about a plain full-stack
loop; IFM's Ouro-style baseline already runs four passes and Huginn-style runs eight core iterations
inside a smaller envelope, so the two reports are sweeping different variables and shouldn't be read as
agreeing or disagreeing on "how many loops."

There's a second, more mechanical echo. [LOTUS](/articles/lotus-latent-reasoning) — the site's other
looped-transformer piece — already does something IFM's ablation independently flags as one of the two
levers that matter: every LOTUS iteration recomputes $\mathbf e + \mathbf h^{(t-1)}$, persistently
feeding the fixed input embeddings back into the loop rather than only conditioning on them once. That's
a specific instance of the same principle behind IFM's write operator $D(\mathbf z_t, \mathbf e)$: keep
re-showing the loop its input. LOTUS applies that idea inside a frozen-backbone latent-reasoning setup
at inference time rather than IFM's from-scratch pretraining setup, so the two aren't directly
comparable — but it's a second, independent place persistent input injection shows up as doing real
work.

## What to trust, and what to hold loosely

<Callout type="warn">
**The scope, precisely.** Every result above is at 730M dense / 8B-resident MoE scale — there is no
evidence yet these findings hold at frontier (100B+) scale, and the report says so. The random-init
result is explicitly narrower than "random init never helps": IFM's own caveat is that these evaluations
"do not directly measure multi-start path independence or extrapolation to recurrent depths beyond those
used during training" — the property random init is classically supposed to buy. The H/L null result is
scoped to a *shared-module* hierarchy only; a closer HRM-Text replica with separate modules is still
under evaluation and not included here. And this whole report has no arXiv listing, lives on a
continuously-updated Notion page billed as "Part I" of a series, and ships with no code yet — treat every
number as provisional until an arXiv version, released code, or a third-party rerun shows up.
</Callout>

## The take

The intuitive story about looped transformers has always centered on the recurrent state itself —
randomize it, and the model is forced to learn something more general. IFM's controlled ablations say
that story is backwards for at least these two designs at this scale: the state's initialization is
close to a wash, sometimes a net loss, and a fashionable H/L hierarchy adds nothing consistent once you
control for everything else. What actually separates a strong looped model from a weak one is much less
exotic — untie a prelude and coda from the loop so the recurrent core can specialize, and keep showing
that core its input at every pass instead of just once. Neither idea needs a random number generator.

That's a genuinely useful result for anyone building a looped model today, and it lines up with what
Nanbeige found the hard way in production: cheap shortcuts inside the loop (shared KV cache, more passes
without restructuring, upcycling instead of retraining) tend to cost you, while spending real compute
and real information on each pass tends to pay. The caveat that matters most is the one IFM states
themselves — this is one group's ablation, at sub-billion-to-8B scale, with no code out yet and no arXiv
paper. "Part I" means there's more coming. Whether the random-init myth holds at 100B+ parameters, and
whether a properly separate-module HRM-Text hierarchy fares better than the shared one tested here, are
open questions the authors name as future work, not settled ones.

---

*Source: [Towards Looped Models Done Right — Part I: Topology, Input Injection, Recurrent-State
Design](https://ifm-research.notion.site/Towards-Looped-Models-Done-Right-3ade511912ec8128987dfeb7a5580043)
(Huang, Shi, Chen, Wen, Liu, Xing, Ma; Institute of Foundation Models, 2026). All benchmark deltas,
equations, and figures are quoted or reproduced from the report; the matched-depth, construction-path,
and believed-vs-measured diagrams are my own illustrations of the same numbers.*