2026-07-22 · 8 min · llm · agents · small-models · open-weights · explainer
Nanbeige4.2-3B is a compact agentic model — 3B non-embedding parameters (4B total), Apache-2.0, bilingual EN/ZH — from the Nanbeige LLM Lab at Boss Zhipin. Its claim is the one every good small model makes: it performs "well beyond its parameter scale," reporting wins over Qwen3.5-9B and Gemma4-12B across tool-use, office-agent, code-agent, and most reasoning benchmarks. The mechanism behind the claim is the interesting part — a Looped Transformer — and with the technical report now out, the config is no longer a mystery: it's a two-pass loop, pretrained from scratch on 28T tokens, then shaped by a four-stage post-training pipeline. Here's the loop, the efficiency story, and which of the numbers to lean on.
The loop: depth without parameters
A standard transformer buys reasoning depth by stacking more distinct layers, each with its own weights. A looped transformer buys it by running the same layers several times — so effective compute depth is (physical layers × loops), while the parameter count stays at just the physical layers. You pay for depth in FLOPs, not weights.
Both columns do 16 layers of computation. The looped one stores only 8 layers’ worth of weights and runs them 2 times; the standard one pays for all 16 in parameters. Nanbeige ships ×2 — the report finds it keeps ~75% of a standard Transformer’s token efficiency while adding real capacity, and that more passes buy almost nothing but slower, less stable training. Same recurrent-depth bet as LOTUS, traded FLOPs-for-weights.
Three design choices in the report make this more than a slogan:
- Two passes, not more. The report studies the loop count directly and lands on 2 as the sweet spot: it keeps roughly 75% of a standard Transformer's token efficiency while adding real capacity. More passes buy almost nothing and make training slower and less stable — so the model loops exactly twice.
- From scratch beats upcycling. You could pretrain a normal transformer and then convert it into a looped one ("upcycling"). Nanbeige compared both and found training the looped architecture from scratch performs significantly better — the model needs to adapt its representations to repeated layer reuse throughout pretraining, not have the loop bolted on afterward.
- They kept the full KV cache. Looping twice normally doubles the attention compute, so they tried sharing the KV cache across passes to halve it. It consistently underperformed, so they kept the full, non-sharing loop — a deliberate choice to spend inference memory on quality.
If that recurrent-depth bet sounds familiar, it's the same one as LOTUS, which loops a padded 3B Transformer to reason in its hidden states — and it's a cousin of the architecture-over-scale thesis in Motif 2.6B. You can explore the tradeoff in the widget above; ×2 is what actually ships.
A stronger base to start from
Before any agent training, the looped base model already leads its weight class. Pretrained from scratch on a 28T-token corpus (larger and cleaner than Nanbeige 4.1's, with up-weighted math, code, and synthetic-QA data — and a first taste of agentic trajectories mixed in), Nanbeige4.2-3B-Base beats Qwen3.5-4B-Base, Gemma4-E4B-Base, and its own predecessor on every reported base benchmark: GSM8K 92.7, BBH 81.6, MBPP 67.6, SuperGPQA 35.2, GPQA 53.3. The knowledge gap is the clearest — on MMLU-Pro the 3B looped base outscores a 4B Qwen base by twelve points:
That head start — the loop plus the refined 28T-token mixture — is what the post-training then turns into agentic behavior.
The results, and how to read them
Here's the headline chart from the report: the same benchmark suite against Gemma4 and Qwen3.5, with Nanbeige4.2-3B in teal, essentially topping every agent and code panel and most reasoning ones.
The cleaner way to see the efficiency argument is to put score against parameters directly. On the public code and agent benchmarks the teal point sits up-and-to-the-left — smaller and higher:
On agentic and code benchmarks the teal point is up-and-left of everything: fewer parameters, higher score. On saturated knowledge tests (GPQA) the field bunches, but even there the 3B model leads. “Well beyond its parameter scale” is the whole pitch, and it holds.
Now the honesty pass, because not all of these bars are the same kind of evidence:
- The comparable, public ones are the strongest signal: SWE-Bench Verified 63.6 (vs Qwen3.5-9B 53.1), SWE-Bench Pro 46.9 (vs 33.8), Terminal-Bench 2.0 44.1 (vs 29.2), LiveCodeBench-V6 72.5, HMMT-Feb-2026 82.8. A 3B model at 63.6 on SWE-Bench Verified is genuinely notable if it holds up in third-party harnesses. The report evaluates everyone under standardized protocols (Table 3), which is a stronger footing than a model card's loose bars.
- Treat the eye-catching ones with care. GPQA-Diamond 87.4 for a 3B model is near-frontier and surprising — it's self-reported in Think mode, the regime where small models gain the most and where contamination is hardest to rule out. Several agent scores (GDPval, AgentIF-Oneday, OfficeQA-Pro) run through Nanbeige's own agent stack, and Recruit-Bench is Nanbeige's in-house benchmark — all reasonable to publish, none of it fully apples-to-apples.
- Where it doesn't lead: Gemma4-12B still wins SciCode (38.2 vs 35.6), IF-Bench (73.5 vs 54.6), and Recruit-Bench (69.4 vs 63.3) — strict instruction-following and some scientific coding aren't the strong suit. The report is upfront about this: best on five of six reasoning benchmarks, not all of them.
Training: synthesize the environments, reward the process
The report's four-stage post-training pipeline is where the agent behavior is actually built.
1. SFT with a STEM-to-agentic curriculum. Starting from the pretrained checkpoint, supervised fine-tuning runs in three stages that stretch the context window 64K → 128K → 256K while sliding the mix of target tokens from reasoning toward agentic interaction — think first, then act:
Each row is one SFT stage. The context window grows 64K → 128K → 256K, and the mix of what the loss is computed on slides from reasoning-STEM toward long-horizon agentic data — the model builds a reasoning foundation first, then spends most of the final stage learning to plan, call tools, read execution feedback, and recover from its own mistakes. Bad intermediate turns stay in context but are masked out of the loss, so it learns recovery without being trained to repeat the error.
The trajectories themselves come from large-scale environment synthesis: a repository-to-task pipeline for software engineering (mine real repos, reconstruct a sandboxed container, keep only fail-to-pass verified tasks), a hybrid real-plus-simulated pipeline for tool use (live MCP servers, Python-reconstructed APIs, and LLM-simulated virtual tools), and an artifact-centric pipeline for office cowork (reports, slides, spreadsheets). Crucially, the same task is solved by multiple heterogeneous scaffolds — Claude Code, OpenHands, SWE-agent, Codex-style drivers — so the model learns scaffold-invariant repair strategies rather than the quirks of one harness. A turn-level loss mask keeps bad intermediate turns in context but out of the loss, so the model learns to recover from mistakes without being trained to repeat them.
2. Two-stage RLHF for hybrid thinking. A pointwise reward model cleans up the failure modes a small model is prone to — repetitive reasoning, cyclic reflection, delayed termination, malformed output. The report's interesting finding is that this general-purpose RLHF generalizes two ways: cross-task (fixing repetition and formatting also lifts math, code, and agentic scores — many agent failures are generation loops, not reasoning errors) and cross-mode (behavior learned on Non-Think responses transfers to Think mode). It's RLHF doing more than safety and style.
3. Length-controlled reasoning RL. A difficulty-aware penalty discourages over-long reasoning on problems the model already solves reliably, while leaving still-hard problems room to explore — cutting tokens without trading away correctness.
4. Agentic RL with action-centric rubrics. Finally, outcome rewards are combined with process rewards — per-turn rubrics scoring tool-call accuracy and the information gained each step — for denser credit assignment over long trajectories. For a model this small, the report finds it more stable to run agentic RL on easier tasks (short trajectories, higher pass@8) than on the hardest ones. Across the RL pipeline, accuracy rises while output tokens fall (e.g. AA-LCR 50.0 → 58.7 with average length dropping 19.5k → 6.7k tokens; PinchBench-V2 55.9 → 74.7).
Small enough to live on your laptop
The payoff is deployment. At 3B non-embedding params the model is meant to run locally — the card
ships recipes for vLLM, SGLang, llama.cpp/GGUF, and Ollama (including MLX on Apple silicon), with a
configurable thinking mode (enable_thinking, preserve_thinking) and XML-format tool calls. Under the
OpenClaw agent framework — evaluated with the same scaffold and tools for every model — Nanbeige
reports beating both Qwen3.5-4B and 9B across all six daily, office, and deep-research benchmarks, with
the widest gaps on office workflows (GDPval 68.8 vs 38.0, AgentIF-Oneday 58.9 vs 32.1). The pitch is a
private, on-device assistant that can still carry multi-step tool workflows.
The take
Nanbeige4.2-3B is another data point for a thesis this site keeps returning to: architecture and training, not raw scale, are increasingly what a small model needs to punch up a weight class — the same lesson as LOTUS, Motif 2.6B, and the sub-1B security models in Antares. The looped-transformer bet is the genuinely interesting bit, and now that the report shows the working — two passes, trained from scratch, full KV cache — it reads less like a marketing line and more like a set of measured tradeoffs. The public code-agent numbers are strong enough to take seriously; just keep the in-house scaffolds and self-reported reasoning scores in the "promising, pending third-party replication" column — which is exactly where an open-weight release lets anyone go check.
Source: the Nanbeige4.2-3B technical report (Nanbeige LLM Lab, 2026) and the model card. Evaluations are self-reported, largely in Think mode, some using in-house scaffolds and benchmarks. The performance figure is Nanbeige's; the interactives are mine.