# Virtual logic depth: looping buys reasoning, not knowledge
> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/virtual-logic-depth
> date: 2026-09-08
> tags: llm, looped-transformers, recurrent-depth, scaling-laws, architecture, explainer
This site has already covered looped transformers from the design side: [Towards Looped Models Done
Right](/articles/looped-models-done-right) walks through IFM Research's ablations isolating which of
three tangled design axes — iteration envelope, input injection, recurrent-state init — actually
separates a strong looped model from a weak one. That piece is about *how* to build the loop.
**[Beyond Parameters: Exploring Virtual Logic Depth for Scaling
Laws](https://arxiv.org/abs/2506.18233)** (arXiv 2506.18233v3, Zhu, Zhang, Li, Shi, Duan, Wang, Zhou,
Banerjee, Qin) asks a different question: once you have a loop, what does it actually scale?
Its answer, compressed to one sentence: reusing transformer layers — what the paper calls **virtual
logical depth (VLD)** — barely moves how much a model knows, but substantially moves how well it
reasons, at a fixed parameter count. Read that as a possible fourth scaling axis alongside depth,
width, and parameter count, and the paper's own closing question follows naturally: does pushing
capability further always mean adding parameters, or can some of it come from reusing the ones you
already have?
*(There's also a circulating claim online that OpenAI's Astra model uses looped transformers
internally. That is an unsourced rumor with no public confirmation — it gets one sentence here and
nothing below depends on it.)*
## The three-line claim, and what each line costs to earn
The abstract states its findings as three numbered claims. Worth quoting directly, because the rest
of this piece is about what had to be true for each one to be measured at all:
1. **Knowledge capacity vs. parameters.** "At a fixed parameter count, VLD leaves knowledge capacity
nearly unchanged (with only minor variance), while across models knowledge capacity scales with
the number of parameters."
2. **Reasoning vs. reuse.** "Properly implemented VLD substantially improves reasoning ability
*without* increasing parameter count, decoupling reasoning from sheer model size."
3. **Robustness and generality.** "The trend of improved reasoning persist across architectures and
configurations."
Claim 1 needs a working definition of "knowledge capacity" precise enough to detect a null result.
Claim 2 needs a working definition of "reasoning" that isn't just memorized benchmark answers. Claim 3
needs more than one architecture. All three sections below exist to check what "properly implemented,"
"nearly unchanged," and "persists across architectures" actually cashed out to.
## How "knowledge capacity" is measured
This is the crux, so it's worth being exact. The paper does **not** use Allen-Zhu & Li's
"physics of language models" bits-per-parameter methodology directly — it cites that work (arXiv
2404.05405) for the *idea* that knowledge capacity is measurable at all, and for the "~2 bits per
parameter" figure it uses to size its experiment, but the actual protocol here is the authors' own,
simpler construction: absorbed information entropy on a synthetic memorization task.
Construct a sequence of $k$ tokens, each drawn i.i.d. uniformly from $n$ possible values (here
$n = 50257$, the tokenizer's vocabulary size, and $k = 640{,}000$). The dataset's total entropy is
fixed by construction:
$$
H_1 = k \log_2 n
$$
Train a small GPT-2 to fit this sequence — literally memorize which random token comes next at each
position, with no validation set, because the entire point is overfitting as hard as possible. Once
training converges, read the model's own softmax output $p_j(x_i)$ at every position and compute:
$$
H_2 = -\sum_{j=1}^{k}\sum_{i=1}^{n} p_j(x_i)\log_2 p_j(x_i)
$$
$H_2$ is the residual entropy the model still can't collapse to zero — the sequence it hasn't
memorized. The difference,
$$
\Delta H = H_1 - H_2
$$
is the paper's knowledge-capacity number: bits of information the model actually absorbed. The
knowledge-capacity experiments use their own small model family — 4-layer GPT-2s at 5M, 10M, 15M, and
20M parameters (hidden size 92 and 184 respectively, matching Allen-Zhu & Li's own settings) — trained
on 8×A100s with Adam at lr $2\times10^{-5}$, sequence length 768 at train time and 1024 at eval.
Two things worth flagging plainly. First, the probe is **literally random tokens**, not facts about
the world — it measures how many bits of arbitrary information a model can cram into its weights, not
how much it knows about, say, biographies or chemistry the way Allen-Zhu & Li's synthetic-biography
benchmarks do. That's a legitimate and controlled way to measure raw absorption capacity, but "the
model didn't get better at absorbing random tokens" is a narrower claim than "the model didn't get
better at storing knowledge" in the everyday sense, and the paper's own framing sometimes slides
between the two. Second — and this matters for reading Figure 1 below — this knowledge-capacity
protocol was run on the 5M–20M model family, not on the larger 50M–200M family used for the reasoning
experiments. The two are related only by extrapolation, not by directly measuring capacity on the
exact models being scored for reasoning.
## How "reasoning capability" is measured
Two ways, and it matters which one is doing the load-bearing work.
**Primary: a synthetic task, iGSM.** Following Ye et al. (2024), the paper synthesizes grade-school
math problems where a target variable's value must be derived by chaining a sequence of definitions —
difficulty controlled directly by the number of algebra operations required. A shortened example, in
the same style as the paper's:
```
Q: Lungs' Platelets equals Lungs' B Cells. Pleural Cavity's B Cells equals 11 more
than the sum of Lungs' Platelets and Lungs' B Cells. Lungs' B Cells equals 10.
How many B Cells does Pleural Cavity have?
A: Lungs' B Cells = 10. Lungs' Platelets = 10. Pleural Cavity's B Cells = 10 + 10 + 11 = 8 (mod 23).
```
The template space is enormous (the paper cites "more than 90 trillion" possible solution templates),
so a model can't solve these by memorizing training examples — it has to actually chain the
derivation. Training uses problems with up to 15 operations; validation uses problems with *exactly*
15 (in-distribution), plus held-out sets at 20 and 21 operations (out-of-distribution, to check whether
gains generalize to harder problems than the model ever trained on).
**Secondary: real benchmarks, but only on one fine-tuned model.** To check the synthetic result isn't
an artifact of the synthetic task, the paper also fine-tunes LLaMA-3.2-3B-Instruct (3.22B parameters,
LoRA with 6.08M trainable — 0.19%) on a 2.3-billion-token multi-domain SFT corpus, comparing a Base
variant against a Cycle-VLD variant, then scores both on Math500, AIME, GPQA, HumanEval, and MBPP.
So: one purpose-built synthetic benchmark carries essentially all of the controlled, multi-configuration
evidence in the paper; real-world benchmarks appear for exactly one model pair, at one scale, with one
reuse pattern. That's a real generalization check, but a single data point's worth of one — worth
keeping in mind before reading "persists across architectures" as more than "we also tried it once,
elsewhere, and it also worked."
## Three ways to reuse the same layer
VLD is defined narrowly: total effective depth minus native layer count. The architecture itself
never changes — the paper repeats the *same* transformer block with tied weights, following the
parameter-sharing setup from Takase & Kiyono (2021), in one of three patterns:
```python
# base: L distinct blocks, each with its own weights
def forward_base(x, blocks):
h = embed(x)
for block in blocks: # B1, B2, ..., BL -- all different weights
h = block(h)
return unembed(h)
# sequence: repeat each block in place, k times, before moving to the next
def forward_sequence(x, blocks, k):
h = embed(x)
for block in blocks:
for _ in range(k):
h = block(h) # B1,B1,B2,B2,B3,B3, ... at k=2
return unembed(h)
# cycle: repeat the whole stack, k times
def forward_cycle(x, blocks, k):
h = embed(x)
for _ in range(k):
for block in blocks:
h = block(h) # B1,B2,B3,B1,B2,B3, ... at k=2
return unembed(h)
# inverse cycle: alternate direction each pass through the stack
def forward_inverse_cycle(x, blocks, k):
h = embed(x)
for i in range(k):
order = blocks if i % 2 == 0 else list(reversed(blocks))
for block in order:
h = block(h) # B1,B2,B3,B3,B2,B1 at k=2
return unembed(h)
```
Inverse cycle exists for a specific reason, not as a third option to round out a grid: prior work
(Liu et al., 2023; Takase & Kiyono, 2021) found that higher decoder layers tend to see larger gradient
norms during training, implying they need more representational freedom than lower layers. Reusing a
*lower*-layer's weights in a *later* position, the reasoning goes, should hurt less than reusing a
later layer's weights again. Whether that reasoning pays off is answered directly by the results below
— and it's the one place the paper runs a small internal ablation on *where* sharing happens rather
than just *how much*: with an 8-layer Cycle backbone, sharing across all 8 layers scores 62.0%, sharing
only the first 4 scores 54.2% (worse than not looping at all), and sharing only the last 4 scores
64.2% — the best number in the table. Where you loop matters as much as how many times.
## What the numbers actually say
Table 1 is the paper's central result, reproduced in full. "VLD Depth" is the repetition factor
applied to the base layer count; op15/op20/op21 are iGSM accuracy at that many chained operations
(op15 is in-distribution, op20 and op21 are out-of-distribution):
| Pattern | VLD depth | 4L op15 | 4L op20 | 4L op21 | 8L op15 |
|---|---|---|---|---|---|
| Base | – | 46.3 | 21.8 | 21.2 | 60.5 |
| Cycle | ×1 | 54.9 | 26.4 | 26.2 | 62.0 |
| Cycle | ×2 | 62.1 | 30.4 | 35.4 | 63.3 |
| Cycle | ×3 | 61.6 | 30.8 | 32.0 | 61.0 |
| Cycle | ×4 | 65.7 | 39.2 | 33.0 | 66.8 |
| Cycle | ×5 | 70.7 | 43.8 | 40.2 | – |
| Sequence | ×1 | 50.7 | 25.6 | 21.2 | 59.2 |
| Sequence | ×2 | 51.2 | 27.0 | 22.2 | 62.1 |
| Sequence | ×3 | 55.5 | 28.0 | 25.0 | 62.9 |
| Inv. Cycle | ×1 | 43.9 | 17.4 | 15.8 | 53.3 |
| Inv. Cycle | ×2 | 50.8 | 24.2 | 18.2 | 62.0 |
| Inv. Cycle | ×3 | 54.8 | 25.2 | 22.2 | 62.8 |
Three things jump out. Cycle wins essentially everywhere it's tested against Sequence and Inverse
Cycle — the "reuse lower layers in higher positions" intuition behind Inverse Cycle doesn't pay off in
this setup; it's the *worst* pattern at every depth on op15. Gains hold up out-of-distribution: at
op20 and op21, harder than anything seen in training, Cycle still climbs with depth. And it isn't
monotonic — the 8-layer backbone dips at Cycle ×3 (61.0, down from 63.3 at ×2) before recovering at
×4, and the paper is upfront that it doesn't yet know why, filing it under future work rather than
smoothing it over.
## The plane the paper is arguing for
Figure 1 is the paper's headline chart, plotting knowledge capacity (x) against reasoning accuracy
(y) for both families at once. Below is a redrawn, interactive version — same data, plain linear axes
instead of the paper's compressed one (see the component's own note for exactly which values are
pixel-measured versus stated verbatim in the text):
The two numbers the paper's own prose calls out are the cleanest version of the claim: a 150M native
model reaches 61.15% accuracy; a 50M model with Cycle VLD applied reaches 62.05% — beating a model
three times its size, at zero added parameters. That's claim 2, concretely. Claim 1 is the part that's
easier to miss: notice that the green points don't sit meaningfully to the *right* of their same-sized
blue counterparts. If VLD were secretly buying knowledge capacity too, the green points would drift
right as well as up. They don't.
## Does it generalize past synthetic math?
Partially, and with a caveat worth stating plainly. Table 2, in full — LLaMA-3.2-3B-Instruct, Base vs.
Cycle-VLD, both LoRA-fine-tuned on the same 2.3B-token corpus from the same pretrained weights:
| Model | Math500 | GPQA | AIME | HumanEval (pass@1) | MBPP (pass@1) |
|---|---|---|---|---|---|
| Base | 30.40 | 29.80 | 3.33 | 37.79 | 38.36 |
| Cycle VLD | 35.40 | 32.32 | 6.67 | 39.52 | 40.22 |
Every column improves, including domains that never appeared in the synthetic experiments — GPQA
science questions and two code-generation benchmarks. That's genuinely useful corroboration that the
effect isn't an artifact of iGSM specifically.
The caveat: this is the one place in the paper where VLD is applied to an *already-pretrained* model
— "the Cycle VLD variant incorporating the layer repetition pattern before training commences," on
top of "identical pretrained weights" shared with the Base variant. In other words, this is upcycling
a non-looped model into a looped one, then LoRA-fine-tuning both variants — not training a looped
architecture from scratch the way every GPT-2 experiment above does. [Nanbeige4.2-3B's technical
report](/articles/nanbeige-4-2-3b), covered on this site as part of [the IFM
piece](/articles/looped-models-done-right), found in production that training a looped architecture
from scratch clearly beat upcycling a pretrained feedforward model into one. VLD's only real-scale
result runs the upcycled path anyway, and it still worked — which either means the upcycling penalty
Nanbeige found doesn't generalize to this setup, or that VLD's real-world gain would be even larger
trained from scratch. The paper doesn't test the from-scratch version at this scale, so which one is
true is genuinely unknown.
## Knowledge capacity really does stay flat
The mechanism behind claim 1, from the paper's own knowledge-capacity experiment (the 5M–20M model
family, not the 50M–200M reasoning family):
Panel (a) is the unsurprising half — more parameters, more absorbed entropy, roughly the trend anyone
would expect. Panel (b) is the actual finding: hold parameters fixed and vary effective depth from 4
to 16 under any of the three reuse patterns, and absorbed entropy barely moves — each model's curve
sits close to its own starting value the whole way across. Running the same tied weights more times
doesn't give the model anywhere new to put information; there's no additional storage being created,
only additional computation over the storage that already exists.
## Causal intervention, not a cross-family fit
Worth being explicit, since the paper's title invokes "scaling laws" and most scaling-law claims on
this site (see [Skaling](/articles/skaling-law) or [the 2026 scaling-laws
survey](/articles/scaling-laws-2026)) are fits across a family of differently-sized models. This one
is different in kind: the load-bearing evidence for claims 1 and 2 is a **controlled, within-model
intervention** — the same base architecture, same parameter count, same training data and schedule,
run with VLD switched on or off. The 50M model at 46.3% and the 50M model at 62.05% are the *same
underlying 4-layer, 12.5M-params-per-layer backbone*, trained from scratch twice, once looped and once
not. That's a real causal comparison, not a correlation read off a scatter of unrelated models.
The part that *is* a cross-family fit is the diagonal itself — "knowledge scales with parameters" is
established by comparing four differently-sized native models (5M/10M/15M/20M for the entropy
experiment, 50M/100M/150M/200M for the reasoning one), the same way any scaling-law paper establishes
a trend line. So the paper makes one causal claim (VLD does not add knowledge capacity, at fixed
params) and one correlational one (knowledge capacity grows with parameter count) — and it's worth
noticing that only the first is actually novel; the second is exactly what every scaling law since
Kaplan already says.
One more scope note on "a new scaling axis": the paper reports **no fitted functional form** anywhere
— no equation relating accuracy or loss to loop count the way Chinchilla's $L(N,D)$ or Skaling's
$L(N,D) = (A/N^\alpha + B/D^\beta)^k + E$ relate loss to parameters and data. Every result above is a
table or a scatter plot of measured points, not a fitted curve extrapolated beyond them. That's a
legitimate way to report a controlled experiment, but it means "scaling law" in this paper's title is
closer to "a scaling *behavior*, demonstrated" than to a fitted law of the Kaplan/Chinchilla/Skaling
form — there's no exponent here to check against a held-out regime the way Skaling's $k=0.41$ gets
checked against far-extrapolation runs.
## The question the paper never asks: matched in what?
This is the single most important methodological gap, and it's worth stating as plainly as the paper
does not: **every comparison in this paper is parameter-matched, and the word "FLOP" does not appear
anywhere in it.** VLD is parameter-matched by construction — that's the entire point of tying weights
across repeated layers, "the actual number of parameters do not change" — but looping a block $k$
times means running that block's forward pass $k$ times per token. Compute per token rises linearly
with the loop factor even while parameters sit perfectly still.
This doesn't invalidate the paper's claims — parameter-matched is a real and useful thing to hold
fixed, and it's exactly what makes the knowledge-capacity result clean (there's no confound from "the
looped model also just has more weights"). But it does mean the framing of VLD as a "free" way to buy
reasoning — free in the sense that no new parameters are needed — is only free on one specific ledger.
On the ledger that actually determines inference latency and serving cost, a Cycle-VLD-×5 model is
paying five times the forward-pass compute of its base backbone for that 70.7% number, and the paper
never puts that number next to the accuracy gain to let a reader weigh one against the other. Anyone
citing "smaller model, better reasoning, no extra parameters" should say "no extra parameters, more
compute per token" in the same breath.
## How many models, over what ranges
Counting from the tables and appendices rather than the prose summary, since "systematic study" claims
are only as strong as the count behind them:
- **Knowledge-capacity experiment**: 4 native baselines (5M, 10M, 15M, 20M params), plus VLD applied to
the smallest and largest of those across 3 patterns × 3 repetition factors — 18 more configurations,
roughly 22 trained models total. Vocabulary $n=50257$, sequence length $k=640{,}000$ tokens per run.
- **Reasoning experiment (pretraining)**: 4 native baselines (4/8/12/16 layers, ~50M/100M/150M/200M
params), plus VLD on the 4-layer and 8-layer backbones across 3 patterns × factors 1–3 (Cycle also
tested to ×5) — Table 1 alone reports 23 distinct rows across both backbones. 500K synthetic iGSM
problems, 8×A100s, trained to convergence.
- **Reasoning experiment (post-training)**: 2 models — LLaMA-3.2-3B-Instruct Base and Cycle-VLD,
LoRA-fine-tuned on 2.3B tokens.
That's on the order of **45–50 trained model configurations** in total, almost all of them GPT-2s
under 200M parameters. It's a real sweep — enough points to see the flat-knowledge / rising-reasoning
pattern hold across two backbones, three reuse patterns, and up to five repetition factors — but it's
a sweep at a scale several orders of magnitude below where "does this survive at frontier scale" could
be answered, and the paper doesn't claim otherwise.
## Where this sits next to Towards Looped Models Done Right
[IFM Research's ablations](/articles/looped-models-done-right) and this paper are answering adjacent
but genuinely different questions, at genuinely different scales, with no benchmark or metric in
common — worth being precise about both the overlap and the gap.
**Different axis, different question.** IFM holds the *reuse pattern* fixed (their Ouro/Huginn family
loops the same block a fixed number of times) and varies three architectural knobs instead: whether a
prelude/coda is untied from the loop, whether the input gets re-injected at every pass, and whether the
recurrent state starts from a random draw or the encoded input directly. VLD holds all of that fixed —
there's no prelude/coda split, no explicit input-injection gate, no state randomization anywhere in
this paper — and instead varies *which layers get reused and in what order* (Sequence vs. Cycle vs.
Inverse Cycle). Two papers looping transformers, ablating almost entirely disjoint sets of design
choices.
**Where they rhyme.** IFM's single widest-reaching lever was persistent input injection — re-showing
the loop its input at every pass rather than once. VLD doesn't have an injection mechanism to test, but
its own internal ablation (sharing layers 5–8 vs. 1–4 vs. all 8) points at a structurally similar idea:
which part of the network gets the "fresh" treatment matters more than how many times something loops
in total. IFM's higher-layers-need-more-freedom citation (Takase & Kiyono, 2021) is the *same* citation
VLD uses to justify Inverse Cycle — and in VLD's results, protecting later layers from sharing (the
5-8-only ablation, 64.2%) beats sharing everything (62.0%), which is the same direction as IFM's
"untie enough of the network that the specialized parts stay specialized" finding, even though neither
paper tested the other's exact intervention.
**Where they don't overlap at all.** IFM never measures anything like a knowledge-capacity probe —
their entire evaluation is a ten-benchmark accuracy suite (ARC-C, HellaSwag, MMLU, and so on), so
"looping doesn't add knowledge capacity" is a claim IFM's report is simply silent on, not one it
agrees or disagrees with. And the scales don't overlap either: IFM works at 730M dense / 8B-resident
MoE; VLD's controlled experiments top out at 200M, with a single 3.2B fine-tune as the only larger
check. Read together, the honest summary is: two independent groups, using non-overlapping methods,
both land on "the loop is doing something real that plain scaling doesn't buy you the same way" — but
neither has replicated the other's specific finding, and VLD's specific "reasoning without knowledge"
framing is, so far, this paper's claim alone.
## What to trust, and what to hold loosely
**Scope, precisely.** The controlled, from-scratch part of this paper — every result that isn't Table
2 — tops out at 200M parameters, several orders of magnitude below frontier scale, and nothing here
tests whether the knowledge/reasoning split holds up there. The knowledge probe is synthetic random-
token memorization, not real-world facts, and it was run on a different (smaller) model family than
the one scored for reasoning — Figure 1's x-axis values for the 50M–200M reasoning models are read
from the chart, not tabulated in text. The one real-scale generalization check (LLaMA-3.2-3B) applies
VLD by upcycling already-pretrained weights rather than training looped from scratch, which is exactly
the shortcut [Nanbeige's production report](/articles/nanbeige-4-2-3b) found underperforms — yet it
still worked here, which is worth noting rather than either dismissing or over-crediting. Every
comparison is parameter-matched only; there is no FLOPs or wall-clock accounting anywhere in the paper.
The code link is still an anonymized review artifact as of the version read here (v3, October 2025),
so nobody outside the author group can yet rerun these numbers.
## The take
The core result survives the scrutiny above better than most single-paper claims do, because the part
that matters most — knowledge capacity staying flat while reasoning accuracy climbs, at fixed
parameters — is a genuine causal intervention on the same architecture, not a correlation dressed up as
one. Cycle beating Sequence and Inverse Cycle at nearly every depth, the out-of-distribution accuracy
holding up at op20/op21, and the same direction of effect showing up again on a real fine-tuned 3B
model are three independent pieces of corroboration pointing the same way.
What doesn't survive as cleanly is the packaging. "A fourth scaling axis" implies a fitted law with an
exponent to check against held-out scale, and this paper doesn't have one — it has a demonstrated
behavior at sub-200M scale plus one 3B spot-check. And "free reasoning gains" implies free, when every
gain in this paper was bought with more forward-pass compute per token, a cost the paper never once
puts a number on. Neither of those gaps makes the central finding wrong. They make it a real, useful,
under-scaled first result — the kind that's worth building on, not the kind that's worth citing as
settled.
---
*Source: [Beyond Parameters: Exploring Virtual Logic Depth for Scaling
Laws](https://arxiv.org/abs/2506.18233) (arXiv 2506.18233v3, Zhu, Zhang, Li, Shi, Duan, Wang, Zhou,
Banerjee, Qin, 12 October 2025, CC BY-SA 4.0), read in full via the arXiv HTML rendering. Table 1,
Table 2, the layer-selection ablation numbers, the entropy equations, and all quoted figures are
reproduced as published; Figure 1 is redrawn (see that component's own provenance note for exactly
which values are exact versus pixel-measured). The compute-accounting figures are an illustrative
first-order estimate, not a number the paper reports itself.*