# Nanbeige4.2-3B: looping a small model up to a big one's depth
> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/nanbeige-4-2-3b
> date: 2026-07-22
> tags: llm, agents, small-models, open-weights, explainer
[Nanbeige4.2-3B](https://huggingface.co/Nanbeige/Nanbeige4.2-3B) is a compact agentic model — **3B
non-embedding parameters** (4B total), Apache-2.0, bilingual EN/ZH — from the Nanbeige LLM Lab at Boss
Zhipin. Its claim is the one every good small model makes: it performs "well beyond its parameter
scale," reporting wins over **Qwen3.5-9B** and **Gemma4-12B** across tool-use, office-agent, code-agent,
and most reasoning benchmarks. The mechanism behind the claim is the interesting part — a **Looped
Transformer** — and with the [technical report](https://huggingface.co/Nanbeige/Nanbeige4.2-3B/blob/main/Nanbeige42_report.pdf)
now out, the config is no longer a mystery: it's a two-pass loop, **pretrained from scratch on 28T
tokens**, then shaped by a four-stage post-training pipeline. Here's the loop, the efficiency story, and
which of the numbers to lean on.
## The loop: depth without parameters
A standard transformer buys reasoning depth by stacking more distinct layers, each with its own
weights. A **looped** transformer buys it by running the *same* layers several times — so effective
compute depth is (physical layers × loops), while the parameter count stays at just the physical
layers. You pay for depth in FLOPs, not weights.
Three design choices in the report make this more than a slogan:
- **Two passes, not more.** The report studies the loop count directly and lands on **2** as the sweet
spot: it keeps roughly **75% of a standard Transformer's token efficiency** while adding real
capacity. More passes buy almost nothing and make training slower and less stable — so the model
loops exactly twice.
- **From scratch beats upcycling.** You *could* pretrain a normal transformer and then convert it into a
looped one ("upcycling"). Nanbeige compared both and found training the looped architecture from
scratch performs **significantly better** — the model needs to adapt its representations to repeated
layer reuse throughout pretraining, not have the loop bolted on afterward.
- **They kept the full KV cache.** Looping twice normally doubles the attention compute, so they tried
sharing the KV cache across passes to halve it. It consistently underperformed, so they **kept the
full, non-sharing loop** — a deliberate choice to spend inference memory on quality.
If that recurrent-depth bet sounds familiar, it's the same one as [LOTUS](/articles/lotus-latent-reasoning),
which loops a padded 3B Transformer to reason in its hidden states — and it's a cousin of the
architecture-over-scale thesis in [Motif 2.6B](/articles/motif-2-6b). You can explore the tradeoff in
the widget above; ×2 is what actually ships.
## A stronger base to start from
Before any agent training, the looped base model already leads its weight class. Pretrained from scratch
on a 28T-token corpus (larger and cleaner than Nanbeige 4.1's, with up-weighted math, code, and
synthetic-QA data — and a first taste of agentic trajectories mixed in), **Nanbeige4.2-3B-Base** beats
Qwen3.5-4B-Base, Gemma4-E4B-Base, and its own predecessor on *every* reported base benchmark: GSM8K
**92.7**, BBH **81.6**, MBPP **67.6**, SuperGPQA **35.2**, GPQA **53.3**. The knowledge gap is the
clearest — on MMLU-Pro the 3B looped base outscores a 4B Qwen base by twelve points:
That head start — the loop plus the refined 28T-token mixture — is what the post-training then turns into
agentic behavior.
## The results, and how to read them
Here's the headline chart from the report: the same benchmark suite against Gemma4 and Qwen3.5, with
Nanbeige4.2-3B in teal, essentially topping every agent and code panel and most reasoning ones.
The cleaner way to see the efficiency argument is to put score against parameters directly. On the
public code and agent benchmarks the teal point sits up-and-to-the-left — smaller *and* higher:
Now the honesty pass, because not all of these bars are the same kind of evidence:
- **The comparable, public ones** are the strongest signal: **SWE-Bench Verified 63.6** (vs Qwen3.5-9B
53.1), **SWE-Bench Pro 46.9** (vs 33.8), **Terminal-Bench 2.0 44.1** (vs 29.2), **LiveCodeBench-V6
72.5**, **HMMT-Feb-2026 82.8**. A 3B model at 63.6 on SWE-Bench Verified is genuinely notable if it
holds up in third-party harnesses. The report evaluates everyone under standardized protocols
(Table 3), which is a stronger footing than a model card's loose bars.
- **Treat the eye-catching ones with care.** GPQA-Diamond **87.4** for a 3B model is near-frontier and
surprising — it's self-reported in Think mode, the regime where small models gain the most and where
contamination is hardest to rule out. Several agent scores (GDPval, AgentIF-Oneday, OfficeQA-Pro) run
through Nanbeige's own agent stack, and **Recruit-Bench is Nanbeige's in-house benchmark** — all
reasonable to publish, none of it fully apples-to-apples.
- **Where it doesn't lead:** Gemma4-12B still wins **SciCode** (38.2 vs 35.6), **IF-Bench** (73.5 vs
54.6), and **Recruit-Bench** (69.4 vs 63.3) — strict instruction-following and some scientific coding
aren't the strong suit. The report is upfront about this: best on five of six reasoning benchmarks,
not all of them.
## Training: synthesize the environments, reward the process
The report's four-stage post-training pipeline is where the agent behavior is actually built.
**1. SFT with a STEM-to-agentic curriculum.** Starting from the pretrained checkpoint, supervised
fine-tuning runs in three stages that stretch the context window 64K → 128K → 256K while sliding the
mix of target tokens from reasoning toward agentic interaction — think first, then act:
The trajectories themselves come from large-scale environment *synthesis*: a repository-to-task pipeline
for software engineering (mine real repos, reconstruct a sandboxed container, keep only fail-to-pass
verified tasks), a hybrid real-plus-simulated pipeline for tool use (live MCP servers, Python-reconstructed
APIs, and LLM-simulated virtual tools), and an artifact-centric pipeline for office cowork (reports,
slides, spreadsheets). Crucially, the same task is solved by **multiple heterogeneous scaffolds** —
Claude Code, OpenHands, SWE-agent, Codex-style drivers — so the model learns scaffold-*invariant* repair
strategies rather than the quirks of one harness. A **turn-level loss mask** keeps bad intermediate turns
in context but out of the loss, so the model learns to recover from mistakes without being trained to
repeat them.
**2. Two-stage RLHF for hybrid thinking.** A pointwise reward model cleans up the failure modes a small
model is prone to — repetitive reasoning, cyclic reflection, delayed termination, malformed output. The
report's interesting finding is that this general-purpose RLHF **generalizes two ways**: *cross-task*
(fixing repetition and formatting also lifts math, code, and agentic scores — many agent failures are
generation loops, not reasoning errors) and *cross-mode* (behavior learned on Non-Think responses
transfers to Think mode). It's RLHF doing more than safety and style.
**3. Length-controlled reasoning RL.** A difficulty-aware penalty discourages over-long reasoning on
problems the model already solves reliably, while leaving still-hard problems room to explore — cutting
tokens without trading away correctness.
**4. Agentic RL with action-centric rubrics.** Finally, outcome rewards are combined with **process
rewards** — per-turn rubrics scoring tool-call accuracy and the information gained each step — for denser
credit assignment over long trajectories. For a model this small, the report finds it more stable to run
agentic RL on *easier* tasks (short trajectories, higher pass@8) than on the hardest ones. Across the RL
pipeline, accuracy rises while output tokens fall (e.g. AA-LCR 50.0 → 58.7 with average length dropping
19.5k → 6.7k tokens; PinchBench-V2 55.9 → 74.7).
## Small enough to live on your laptop
The payoff is deployment. At 3B non-embedding params the model is meant to run **locally** — the card
ships recipes for vLLM, SGLang, `llama.cpp`/GGUF, and Ollama (including MLX on Apple silicon), with a
configurable thinking mode (`enable_thinking`, `preserve_thinking`) and XML-format tool calls. Under the
**OpenClaw** agent framework — evaluated with the *same* scaffold and tools for every model — Nanbeige
reports beating both Qwen3.5-4B and 9B across all six daily, office, and deep-research benchmarks, with
the widest gaps on office workflows (GDPval 68.8 vs 38.0, AgentIF-Oneday 58.9 vs 32.1). The pitch is a
private, on-device assistant that can still carry multi-step tool workflows.
## The take
Nanbeige4.2-3B is another data point for a thesis this site keeps returning to: **architecture and
training, not raw scale, are increasingly what a small model needs to punch up a weight class** — the
same lesson as [LOTUS](/articles/lotus-latent-reasoning), [Motif 2.6B](/articles/motif-2-6b), and the
sub-1B security models in [Antares](/articles/antares). The looped-transformer bet is the genuinely
interesting bit, and now that the report shows the working — two passes, trained from scratch, full KV
cache — it reads less like a marketing line and more like a set of measured tradeoffs. The public
code-agent numbers are strong enough to take seriously; just keep the in-house scaffolds and
self-reported reasoning scores in the "promising, pending third-party replication" column — which is
exactly where an open-weight release lets anyone go check.
---
*Source: the [Nanbeige4.2-3B technical report](https://huggingface.co/Nanbeige/Nanbeige4.2-3B/blob/main/Nanbeige42_report.pdf)
(Nanbeige LLM Lab, 2026) and the [model card](https://huggingface.co/Nanbeige/Nanbeige4.2-3B).
Evaluations are self-reported, largely in Think mode, some using in-house scaffolds and benchmarks. The
performance figure is Nanbeige's; the interactives are mine.*