Virtual logic depth: looping buys reasoning, not knowledge

This site has already covered looped transformers from the design side: Towards Looped Models Done Right walks through IFM Research's ablations isolating which of three tangled design axes — iteration envelope, input injection, recurrent-state init — actually separates a strong looped model from a weak one. That piece is about how to build the loop. Beyond Parameters: Exploring Virtual Logic Depth for Scaling Laws (arXiv 2506.18233v3, Zhu, Zhang, Li, Shi, Duan, Wang, Zhou, Banerjee, Qin) asks a different question: once you have a loop, what does it actually scale?

Its answer, compressed to one sentence: reusing transformer layers — what the paper calls virtual logical depth (VLD) — barely moves how much a model knows, but substantially moves how well it reasons, at a fixed parameter count. Read that as a possible fourth scaling axis alongside depth, width, and parameter count, and the paper's own closing question follows naturally: does pushing capability further always mean adding parameters, or can some of it come from reusing the ones you already have?

(There's also a circulating claim online that OpenAI's Astra model uses looped transformers internally. That is an unsourced rumor with no public confirmation — it gets one sentence here and nothing below depends on it.)

The three-line claim, and what each line costs to earn

The abstract states its findings as three numbered claims. Worth quoting directly, because the rest of this piece is about what had to be true for each one to be measured at all:

1. Knowledge capacity vs. parameters. "At a fixed parameter count, VLD leaves knowledge capacity nearly unchanged (with only minor variance), while across models knowledge capacity scales with the number of parameters."
2. Reasoning vs. reuse. "Properly implemented VLD substantially improves reasoning ability without increasing parameter count, decoupling reasoning from sheer model size."
3. Robustness and generality. "The trend of improved reasoning persist across architectures and configurations."

Claim 1 needs a working definition of "knowledge capacity" precise enough to detect a null result. Claim 2 needs a working definition of "reasoning" that isn't just memorized benchmark answers. Claim 3 needs more than one architecture. All three sections below exist to check what "properly implemented," "nearly unchanged," and "persists across architectures" actually cashed out to.

How "knowledge capacity" is measured

This is the crux, so it's worth being exact. The paper does not use Allen-Zhu & Li's "physics of language models" bits-per-parameter methodology directly — it cites that work (arXiv 2404.05405) for the idea that knowledge capacity is measurable at all, and for the "~2 bits per parameter" figure it uses to size its experiment, but the actual protocol here is the authors' own, simpler construction: absorbed information entropy on a synthetic memorization task.

Construct a sequence of $k$ tokens, each drawn i.i.d. uniformly from $n$ possible values (here $n = 50257$, the tokenizer's vocabulary size, and $k = 640{,}000$). The dataset's total entropy is fixed by construction:

Train a small GPT-2 to fit this sequence — literally memorize which random token comes next at each position, with no validation set, because the entire point is overfitting as hard as possible. Once training converges, read the model's own softmax output $p_j(x_i)$ at every position and compute:

$H_2$ is the residual entropy the model still can't collapse to zero — the sequence it hasn't memorized. The difference,

is the paper's knowledge-capacity number: bits of information the model actually absorbed. The knowledge-capacity experiments use their own small model family — 4-layer GPT-2s at 5M, 10M, 15M, and 20M parameters (hidden size 92 and 184 respectively, matching Allen-Zhu & Li's own settings) — trained on 8×A100s with Adam at lr $2\times10^{-5}$, sequence length 768 at train time and 1024 at eval.

Two things worth flagging plainly. First, the probe is literally random tokens, not facts about the world — it measures how many bits of arbitrary information a model can cram into its weights, not how much it knows about, say, biographies or chemistry the way Allen-Zhu & Li's synthetic-biography benchmarks do. That's a legitimate and controlled way to measure raw absorption capacity, but "the model didn't get better at absorbing random tokens" is a narrower claim than "the model didn't get better at storing knowledge" in the everyday sense, and the paper's own framing sometimes slides between the two. Second — and this matters for reading Figure 1 below — this knowledge-capacity protocol was run on the 5M–20M model family, not on the larger 50M–200M family used for the reasoning experiments. The two are related only by extrapolation, not by directly measuring capacity on the exact models being scored for reasoning.

How "reasoning capability" is measured

Two ways, and it matters which one is doing the load-bearing work.

Primary: a synthetic task, iGSM. Following Ye et al. (2024), the paper synthesizes grade-school math problems where a target variable's value must be derived by chaining a sequence of definitions — difficulty controlled directly by the number of algebra operations required. A shortened example, in the same style as the paper's:

Q: Lungs' Platelets equals Lungs' B Cells. Pleural Cavity's B Cells equals 11 more
   than the sum of Lungs' Platelets and Lungs' B Cells. Lungs' B Cells equals 10.
   How many B Cells does Pleural Cavity have?
A: Lungs' B Cells = 10. Lungs' Platelets = 10. Pleural Cavity's B Cells = 10 + 10 + 11 = 8 (mod 23).

The template space is enormous (the paper cites "more than 90 trillion" possible solution templates), so a model can't solve these by memorizing training examples — it has to actually chain the derivation. Training uses problems with up to 15 operations; validation uses problems with exactly 15 (in-distribution), plus held-out sets at 20 and 21 operations (out-of-distribution, to check whether gains generalize to harder problems than the model ever trained on).

Secondary: real benchmarks, but only on one fine-tuned model. To check the synthetic result isn't an artifact of the synthetic task, the paper also fine-tunes LLaMA-3.2-3B-Instruct (3.22B parameters, LoRA with 6.08M trainable — 0.19%) on a 2.3-billion-token multi-domain SFT corpus, comparing a Base variant against a Cycle-VLD variant, then scores both on Math500, AIME, GPQA, HumanEval, and MBPP.

So: one purpose-built synthetic benchmark carries essentially all of the controlled, multi-configuration evidence in the paper; real-world benchmarks appear for exactly one model pair, at one scale, with one reuse pattern. That's a real generalization check, but a single data point's worth of one — worth keeping in mind before reading "persists across architectures" as more than "we also tried it once, elsewhere, and it also worked."

Three ways to reuse the same layer

VLD is defined narrowly: total effective depth minus native layer count. The architecture itself never changes — the paper repeats the same transformer block with tied weights, following the parameter-sharing setup from Takase & Kiyono (2021), in one of three patterns:

# base: L distinct blocks, each with its own weights
def forward_base(x, blocks):
    h = embed(x)
    for block in blocks:                  # B1, B2, ..., BL -- all different weights
        h = block(h)
    return unembed(h)
 
# sequence: repeat each block in place, k times, before moving to the next
def forward_sequence(x, blocks, k):
    h = embed(x)
    for block in blocks:
        for _ in range(k):
            h = block(h)                  # B1,B1,B2,B2,B3,B3, ... at k=2
    return unembed(h)
 
# cycle: repeat the whole stack, k times
def forward_cycle(x, blocks, k):
    h = embed(x)
    for _ in range(k):
        for block in blocks:
            h = block(h)                  # B1,B2,B3,B1,B2,B3, ... at k=2
    return unembed(h)
 
# inverse cycle: alternate direction each pass through the stack
def forward_inverse_cycle(x, blocks, k):
    h = embed(x)
    for i in range(k):
        order = blocks if i % 2 == 0 else list(reversed(blocks))
        for block in order:
            h = block(h)                  # B1,B2,B3,B3,B2,B1 at k=2
    return unembed(h)

expand

Inverse cycle exists for a specific reason, not as a third option to round out a grid: prior work (Liu et al., 2023; Takase & Kiyono, 2021) found that higher decoder layers tend to see larger gradient norms during training, implying they need more representational freedom than lower layers. Reusing a lower-layer's weights in a later position, the reasoning goes, should hurt less than reusing a later layer's weights again. Whether that reasoning pays off is answered directly by the results below — and it's the one place the paper runs a small internal ablation on where sharing happens rather than just how much: with an 8-layer Cycle backbone, sharing across all 8 layers scores 62.0%, sharing only the first 4 scores 54.2% (worse than not looping at all), and sharing only the last 4 scores 64.2% — the best number in the table. Where you loop matters as much as how many times.

What the numbers actually say

Table 1 is the paper's central result, reproduced in full. "VLD Depth" is the repetition factor applied to the base layer count; op15/op20/op21 are iGSM accuracy at that many chained operations (op15 is in-distribution, op20 and op21 are out-of-distribution):

Three things jump out. Cycle wins essentially everywhere it's tested against Sequence and Inverse Cycle — the "reuse lower layers in higher positions" intuition behind Inverse Cycle doesn't pay off in this setup; it's the worst pattern at every depth on op15. Gains hold up out-of-distribution: at op20 and op21, harder than anything seen in training, Cycle still climbs with depth. And it isn't monotonic — the 8-layer backbone dips at Cycle ×3 (61.0, down from 63.3 at ×2) before recovering at ×4, and the paper is upfront that it doesn't yet know why, filing it under future work rather than smoothing it over.

The plane the paper is arguing for

Figure 1 is the paper's headline chart, plotting knowledge capacity (x) against reasoning accuracy (y) for both families at once. Below is a redrawn, interactive version — same data, plain linear axes instead of the paper's compressed one (see the component's own note for exactly which values are pixel-measured versus stated verbatim in the text):

expand

The two numbers the paper's own prose calls out are the cleanest version of the claim: a 150M native model reaches 61.15% accuracy; a 50M model with Cycle VLD applied reaches 62.05% — beating a model three times its size, at zero added parameters. That's claim 2, concretely. Claim 1 is the part that's easier to miss: notice that the green points don't sit meaningfully to the right of their same-sized blue counterparts. If VLD were secretly buying knowledge capacity too, the green points would drift right as well as up. They don't.

Does it generalize past synthetic math?

Partially, and with a caveat worth stating plainly. Table 2, in full — LLaMA-3.2-3B-Instruct, Base vs. Cycle-VLD, both LoRA-fine-tuned on the same 2.3B-token corpus from the same pretrained weights:

Every column improves, including domains that never appeared in the synthetic experiments — GPQA science questions and two code-generation benchmarks. That's genuinely useful corroboration that the effect isn't an artifact of iGSM specifically.

The caveat: this is the one place in the paper where VLD is applied to an already-pretrained model — "the Cycle VLD variant incorporating the layer repetition pattern before training commences," on top of "identical pretrained weights" shared with the Base variant. In other words, this is upcycling a non-looped model into a looped one, then LoRA-fine-tuning both variants — not training a looped architecture from scratch the way every GPT-2 experiment above does. Nanbeige4.2-3B's technical report, covered on this site as part of the IFM piece, found in production that training a looped architecture from scratch clearly beat upcycling a pretrained feedforward model into one. VLD's only real-scale result runs the upcycled path anyway, and it still worked — which either means the upcycling penalty Nanbeige found doesn't generalize to this setup, or that VLD's real-world gain would be even larger trained from scratch. The paper doesn't test the from-scratch version at this scale, so which one is true is genuinely unknown.

Knowledge capacity really does stay flat

The mechanism behind claim 1, from the paper's own knowledge-capacity experiment (the 5M–20M model family, not the 50M–200M reasoning family):

expand

Panel (a) is the unsurprising half — more parameters, more absorbed entropy, roughly the trend anyone would expect. Panel (b) is the actual finding: hold parameters fixed and vary effective depth from 4 to 16 under any of the three reuse patterns, and absorbed entropy barely moves — each model's curve sits close to its own starting value the whole way across. Running the same tied weights more times doesn't give the model anywhere new to put information; there's no additional storage being created, only additional computation over the storage that already exists.

Causal intervention, not a cross-family fit

Worth being explicit, since the paper's title invokes "scaling laws" and most scaling-law claims on this site (see Skaling or the 2026 scaling-laws survey) are fits across a family of differently-sized models. This one is different in kind: the load-bearing evidence for claims 1 and 2 is a controlled, within-model intervention — the same base architecture, same parameter count, same training data and schedule, run with VLD switched on or off. The 50M model at 46.3% and the 50M model at 62.05% are the same underlying 4-layer, 12.5M-params-per-layer backbone, trained from scratch twice, once looped and once not. That's a real causal comparison, not a correlation read off a scatter of unrelated models.

The part that is a cross-family fit is the diagonal itself — "knowledge scales with parameters" is established by comparing four differently-sized native models (5M/10M/15M/20M for the entropy experiment, 50M/100M/150M/200M for the reasoning one), the same way any scaling-law paper establishes a trend line. So the paper makes one causal claim (VLD does not add knowledge capacity, at fixed params) and one correlational one (knowledge capacity grows with parameter count) — and it's worth noticing that only the first is actually novel; the second is exactly what every scaling law since Kaplan already says.

One more scope note on "a new scaling axis": the paper reports no fitted functional form anywhere — no equation relating accuracy or loss to loop count the way Chinchilla's $L(N,D)$ or Skaling's $L(N,D) = (A/N^\alpha + B/D^\beta)^k + E$ relate loss to parameters and data. Every result above is a table or a scatter plot of measured points, not a fitted curve extrapolated beyond them. That's a legitimate way to report a controlled experiment, but it means "scaling law" in this paper's title is closer to "a scaling behavior, demonstrated" than to a fitted law of the Kaplan/Chinchilla/Skaling form — there's no exponent here to check against a held-out regime the way Skaling's $k=0.41$ gets checked against far-extrapolation runs.

The question the paper never asks: matched in what?

This is the single most important methodological gap, and it's worth stating as plainly as the paper does not: every comparison in this paper is parameter-matched, and the word "FLOP" does not appear anywhere in it. VLD is parameter-matched by construction — that's the entire point of tying weights across repeated layers, "the actual number of parameters do not change" — but looping a block $k$ times means running that block's forward pass $k$ times per token. Compute per token rises linearly with the loop factor even while parameters sit perfectly still.

This doesn't invalidate the paper's claims — parameter-matched is a real and useful thing to hold fixed, and it's exactly what makes the knowledge-capacity result clean (there's no confound from "the looped model also just has more weights"). But it does mean the framing of VLD as a "free" way to buy reasoning — free in the sense that no new parameters are needed — is only free on one specific ledger. On the ledger that actually determines inference latency and serving cost, a Cycle-VLD-×5 model is paying five times the forward-pass compute of its base backbone for that 70.7% number, and the paper never puts that number next to the accuracy gain to let a reader weigh one against the other. Anyone citing "smaller model, better reasoning, no extra parameters" should say "no extra parameters, more compute per token" in the same breath.

How many models, over what ranges

Counting from the tables and appendices rather than the prose summary, since "systematic study" claims are only as strong as the count behind them:

• Knowledge-capacity experiment: 4 native baselines (5M, 10M, 15M, 20M params), plus VLD applied to the smallest and largest of those across 3 patterns × 3 repetition factors — 18 more configurations, roughly 22 trained models total. Vocabulary $n=50257$, sequence length $k=640{,}000$ tokens per run.
• Reasoning experiment (pretraining): 4 native baselines (4/8/12/16 layers, ~50M/100M/150M/200M params), plus VLD on the 4-layer and 8-layer backbones across 3 patterns × factors 1–3 (Cycle also tested to ×5) — Table 1 alone reports 23 distinct rows across both backbones. 500K synthetic iGSM problems, 8×A100s, trained to convergence.
• Reasoning experiment (post-training): 2 models — LLaMA-3.2-3B-Instruct Base and Cycle-VLD, LoRA-fine-tuned on 2.3B tokens.

That's on the order of 45–50 trained model configurations in total, almost all of them GPT-2s under 200M parameters. It's a real sweep — enough points to see the flat-knowledge / rising-reasoning pattern hold across two backbones, three reuse patterns, and up to five repetition factors — but it's a sweep at a scale several orders of magnitude below where "does this survive at frontier scale" could be answered, and the paper doesn't claim otherwise.

Where this sits next to Towards Looped Models Done Right

IFM Research's ablations and this paper are answering adjacent but genuinely different questions, at genuinely different scales, with no benchmark or metric in common — worth being precise about both the overlap and the gap.

Different axis, different question. IFM holds the reuse pattern fixed (their Ouro/Huginn family loops the same block a fixed number of times) and varies three architectural knobs instead: whether a prelude/coda is untied from the loop, whether the input gets re-injected at every pass, and whether the recurrent state starts from a random draw or the encoded input directly. VLD holds all of that fixed — there's no prelude/coda split, no explicit input-injection gate, no state randomization anywhere in this paper — and instead varies which layers get reused and in what order (Sequence vs. Cycle vs. Inverse Cycle). Two papers looping transformers, ablating almost entirely disjoint sets of design choices.

Where they rhyme. IFM's single widest-reaching lever was persistent input injection — re-showing the loop its input at every pass rather than once. VLD doesn't have an injection mechanism to test, but its own internal ablation (sharing layers 5–8 vs. 1–4 vs. all 8) points at a structurally similar idea: which part of the network gets the "fresh" treatment matters more than how many times something loops in total. IFM's higher-layers-need-more-freedom citation (Takase & Kiyono, 2021) is the same citation VLD uses to justify Inverse Cycle — and in VLD's results, protecting later layers from sharing (the 5-8-only ablation, 64.2%) beats sharing everything (62.0%), which is the same direction as IFM's "untie enough of the network that the specialized parts stay specialized" finding, even though neither paper tested the other's exact intervention.

Where they don't overlap at all. IFM never measures anything like a knowledge-capacity probe — their entire evaluation is a ten-benchmark accuracy suite (ARC-C, HellaSwag, MMLU, and so on), so "looping doesn't add knowledge capacity" is a claim IFM's report is simply silent on, not one it agrees or disagrees with. And the scales don't overlap either: IFM works at 730M dense / 8B-resident MoE; VLD's controlled experiments top out at 200M, with a single 3.2B fine-tune as the only larger check. Read together, the honest summary is: two independent groups, using non-overlapping methods, both land on "the loop is doing something real that plain scaling doesn't buy you the same way" — but neither has replicated the other's specific finding, and VLD's specific "reasoning without knowledge" framing is, so far, this paper's claim alone.

What to trust, and what to hold loosely

The take

The core result survives the scrutiny above better than most single-paper claims do, because the part that matters most — knowledge capacity staying flat while reasoning accuracy climbs, at fixed parameters — is a genuine causal intervention on the same architecture, not a correlation dressed up as one. Cycle beating Sequence and Inverse Cycle at nearly every depth, the out-of-distribution accuracy holding up at op20/op21, and the same direction of effect showing up again on a real fine-tuned 3B model are three independent pieces of corroboration pointing the same way.

What doesn't survive as cleanly is the packaging. "A fourth scaling axis" implies a fitted law with an exponent to check against held-out scale, and this paper doesn't have one — it has a demonstrated behavior at sub-200M scale plus one 3B spot-check. And "free reasoning gains" implies free, when every gain in this paper was bought with more forward-pass compute per token, a cost the paper never once puts a number on. Neither of those gaps makes the central finding wrong. They make it a real, useful, under-scaled first result — the kind that's worth building on, not the kind that's worth citing as settled.

Source: Beyond Parameters: Exploring Virtual Logic Depth for Scaling Laws (arXiv 2506.18233v3, Zhu, Zhang, Li, Shi, Duan, Wang, Zhou, Banerjee, Qin, 12 October 2025, CC BY-SA 4.0), read in full via the arXiv HTML rendering. Table 1, Table 2, the layer-selection ablation numbers, the entropy equations, and all quoted figures are reproduced as published; Figure 1 is redrawn (see that component's own provenance note for exactly which values are exact versus pixel-measured). The compute-accounting figures are an illustrative first-order estimate, not a number the paper reports itself.