2026-07-13 · 10 min · llm · mixture-of-experts · pretraining · inference-optimization · explainer
Most open-model releases are open in name only: you get weights and an aggregate token count, not the data, recipe, or checkpoints needed to audit or rebuild them. Most general-purpose multilingual models spread their capacity thinly across dozens of languages, leaving German underrepresented relative to its economic weight. And most are dense full-attention Transformers, whose per-sequence KV cache grows with context and drags throughput down exactly in the long-context, high-concurrency regime that costs the most to serve. Soofi S 30B-A3B — from a German consortium coordinated by the KI Bundesverband (DFKI, Fraunhofer IAIS/IIS, TU Darmstadt, and others), funded by the German BMWE — sets out to close all three gaps at once, and to do it on sovereign European infrastructure.
The design that ties those goals together is a hybrid Mamba–Transformer Mixture-of-Experts: 31.6B total parameters, but only ~3.2B active per token, and a backbone that is mostly linear-time Mamba-2 with attention in just 6 of 52 layers. That last choice is the whole serving story — decode throughput stays nearly flat as context grows, where dense baselines fall off a cliff:
The dense 14B line falls off a cliff as context grows — each decoded token re-reads a KV cache that grows with sequence length. Soofi keeps only 6 of 52 layersas attention (~6 KB/token, 11–53× smaller than the dense baselines) while 23 Mamba-2 layers carry a fixed-size recurrent state, so its throughput stays nearly flat and is a reported 9.2× the dense 14B at 40K. Qwen3.5, the other hybrid, decays more gently than dense but still trails Soofi.
The architecture: hybrid backbone, sparse experts
Soofi S reuses the openly published Nemotron 3 Nano reference design without modification — a deliberate choice, so the effect of the German–English data recipe can be measured against an architecture-identical control (the same-arch Nemotron baseline). The backbone is 52 layers: 23 Mamba-2 sequence-mixing layers, 23 granular MoE layers, and only 6 Grouped-Query Attention layers, distributed sparsely through the depth. Scrub the stack — and note how few layers actually hold a cache:
Most of the depth is Mamba-2 (linear-time sequence mixing, a fixed-size recurrent state) and granular MoE capacity. Full attention appears in only 6 of 52 layers — those are the only ones whose cache grows with context. It is the same hybrid recipe as Nemotron, reused deliberately so the German–English data recipe can be measured against an architecture-identical control.
The Mamba-2 layers carry most of the sequence mixing with a fixed-size recurrent state; the attention layers give exact long-range recall but are the only ones whose cache grows. The capacity lives in the MoE layers, and this is where "30B at the cost of 3B" comes from. Each MoE layer has 128 routed experts plus 2 shared experts; a learned, sigmoid-gated router activates just 6 routed experts per token, with the 2 shared always on:
A learned, sigmoid-gated router sends each token to its top-6 of 128 routed experts; 2 shared experts run for every token. Drag to reroute — the chosen set changes token to token, but the count stays fixed. Summed over all 23 MoE layers, that is how a 31.6B-parameter model activates only ~3.2B per token: the capacity of a 30B network at roughly the inference cost of a 3B one.
The exact config, for reproducibility (Table 1): model dimension 2688, 32 attention query heads over just 2 KV heads (head dim 128), Mamba-2 state dimension 128, expert dimension 1856, squared-ReLU MoE activation, RMSNorm, no positional embeddings, untied embeddings. Total 31.6B parameters, ~3.2B active per token (~3.6B including embeddings).
Why the cache stays near-constant
Decoding is memory-bandwidth bound: every generated token must re-read the model weights and the attention cache of every sequence in the batch. In a dense full-attention model that per-sequence cache grows with context, so at tens or hundreds of thousands of tokens, served many-at-once, the KV reads come to dominate and throughput decays. Soofi's hybrid backbone attacks exactly this. Only 6 of 52 layers keep a KV cache, with 2 KV heads each, so the incremental attention-cache footprint is about 6 KB per token per sequence — the report puts that at 11–53× lower than the dense models in its comparison. As context grows, only that small attention component scales with length; the Mamba-2 recurrent state stays constant-size.
The measured payoff: at 40K context and batch 32, Soofi sustains 4.82k aggregate decode TPS/GPU, a reported 9.2× over Ministral 3 14B, while fitting the weights and all 32 sequence states on a single GPU. Across 4K→256K the aggregate decode rate stays essentially flat (no point more than ~34% below the 4K value), where dense throughput decays with context. Among the comparison models only Qwen3.5 — itself a Gated-DeltaNet hybrid — scales similarly, and its 35B-A3B variant still measures ~1.9× slower than Soofi at 40K. The prefill side shows the same shape: time-to-first-token at 256K is 372.7s for Soofi versus 2,058.9s for dense Ministral 3 14B and 6,428.6s for a dense Qwen3 32B control.
The data: ~27T tokens, German on purpose
Soofi S was pretrained on approximately 27 trillion tokens (~26.68T actually consumed) under a three-phase Warmup–Stable–Decay curriculum: ~20T of diverse, quality-tiered pretraining, ~6.58T of high-quality annealing, and a ~0.10T long-context extension that pushes the usable window to 1M tokens. The defining move is that German is deliberately up-weighted — to 7.2% of the stable phase and 15.32% of the annealing mixture, more than triple the ~5% total multilingual share of the reference Nemotron recipe, and concentrated in a single language rather than spread across dozens.
The corpus is documented at the granularity of individual source datasets — raw tokens, epoch multiplier, effective tokens, and even sources that were evaluated and excluded — so the mixture can be audited and, where licenses permit, rebuilt. German coverage combines naturally occurring web and document text (HPLT, German Commons, Genios, German FinePDFs/FineWiki) with machine-translated and synthetic German, since high-quality native German text is far scarcer than English. The report identifies that German data pipeline as the principal bottleneck for further gains.
Just as notable is where it ran. Soofi S was trained end-to-end on the Industrial AI Cloud operated by Deutsche Telekom in Munich — up to 512 NVIDIA B200 GPUs (64 DGX B200 nodes), ~253,000 B200 GPU-hours from 24 March to 13 May 2026, on a facility powered by renewable energy and cooled with water from the Eisbach canal. Training on German soil under European data-protection rules is itself part of the "sovereign" claim.
Results, and how to read them
Against a set of large open-source models (Alia 40B, EuroLLM 22B, Apertus 70B, Olmo 3 32B), Soofi S is the strongest in the set: highest German aggregate and, among fully open models, the highest English and German evaluation scores — ahead of Olmo 3 32B and Apertus 70B despite activating a fraction of their parameters.
The point the authors most want to land is capability-per-active-parameter: Soofi matches dense 14–27B models on English and German aggregates while activating only ~3.2B parameters per token.
It also posts the best code aggregates in that comparison (HumanEval 73.8, MBPP 70.2, HumanEval-DE 65.5, MBPP-DE 84.2 — first on four of five code benchmarks), and leads the set on mathematics (GSM8K 86.1).
Read against the open-weight set, the picture is more measured and the report says so: Soofi is not the top model on aggregate — Qwen3.5 35B-A3B leads (English 74.6, German 81.6), and Soofi's English aggregate (70.1) essentially ties Gemma 3 27B and Ministral 3 14B (both 70.3). Its clearest, cleanest result is the architecture-identical comparison: versus Nemotron 3 Nano 30B-A3B (same backbone, different data), the German–English recipe lifts German aggregate +4.2, held-out English +6.7, GPQA-Diamond +9.6, and German-language proficiency (GLP-DE) +15.1 — while English capability is preserved or improved, the usual price of monolingual specialization avoided.
The honest caveats
The report is candid about its own limitations. Competition-style math in German is the clearest gap to the frontier: Minerva MATH-DE 56.0 trails Qwen3.5 35B-A3B (76.5) and Gemma 3 27B (65.6). Open-domain factual recall is capacity-limited — NaturalQuestions 79.0 trails the largest dense baselines (Gemma 3 27B 83.5), consistent with storing world knowledge in only ~3B active parameters (the authors expect retrieval-augmentation to close this in practice). And on openness itself the report draws an explicit line: it satisfies the OSI's OSAID 1.0 (weights, checkpoints, training and eval code, exact per-source data accounting, all under permissive licenses), but falls short of the stricter "every training token must be redistributable" bar on exactly one component — the commercially licensed Genios corpus (1.3% of Phase 1) — so ~99% of the mixture, not 100%, can be independently reconstructed.
The take
Soofi S's contribution is less a new mechanism than a thesis about deployment cost, executed transparently. The near-constant cache is the load-bearing idea: by keeping only 6 of 52 layers as attention and letting Mamba-2 carry the rest with a fixed-size state, decode throughput stops caring about context length — which is where dense models bleed. Wrap that in a sparse MoE (3.2B active of 31.6B) and you get a model that serves like a 3B but scores like a 14–27B dense on its target languages. Set the knobs honestly — author-reported numbers, an author-defined capability index, a scoped baseline set, a 3B active budget, real gaps in German competition math and factual recall — and what remains is genuinely notable: a fully documented, per-source-audited, German–English pretraining run on sovereign European B200 hardware, released with checkpoints and code. As a template for "open in substance, efficient by architecture, and built at home," it is a clean and unusually legible bet.
Built on the Soofi S Pretraining Report v1.0 ("A Sovereign, Open-Source Foundation Model for German and English", the Soofi-Team; consortium coordinated by the KI Bundesverband, funded by the German BMWE). Architecture, data, and benchmark figures are quoted from the report for commentary; the interactive diagrams are illustrations of the mechanism, and the throughput curves use the report's measured endpoints with an illustrative in-between shape. Related: the architecture-shared Nemotron in NVFP4, mixture-of-experts from scratch, how LLM inference works, and large-scale pretraining.