# Soofi S: a sovereign 3B-active model that keeps its cache near-constant

> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/soofi-s
> date: 2026-07-13
> tags: llm, mixture-of-experts, pretraining, inference-optimization, explainer
Most open-model releases are open in name only: you get weights and an aggregate token count, not the
data, recipe, or checkpoints needed to audit or rebuild them. Most general-purpose multilingual models
spread their capacity thinly across dozens of languages, leaving German underrepresented relative to its
economic weight. And most are dense full-attention Transformers, whose per-sequence [KV
cache](/articles/how-llm-inference-works) grows with context and drags throughput down exactly in the
long-context, high-concurrency regime that costs the most to serve. **Soofi S 30B-A3B** — from a German
consortium coordinated by the KI Bundesverband (DFKI, Fraunhofer IAIS/IIS, TU Darmstadt, and others),
funded by the German BMWE — sets out to close all three gaps at once, and to do it on sovereign European
infrastructure.

The design that ties those goals together is a **hybrid Mamba–Transformer Mixture-of-Experts**: 31.6B
total parameters, but only ~3.2B active per token, and a backbone that is mostly linear-time Mamba-2 with
attention in just 6 of 52 layers. That last choice is the whole serving story — decode throughput stays
nearly flat as context grows, where dense baselines fall off a cliff:

<ThroughputScaling />

<Figure
  src="/articles/soofi-s/fig1.png"
  alt="Two panels. Left: Capability Index versus aggregate decode TPS per GPU at 40K context on a log x-axis; Soofi S 30B-A3B sits top-right (high capability, highest throughput) as a star, above clusters of international and European open models. Right: aggregate decode TPS per GPU versus context length from 4K to 256K; the Soofi S line stays flat near the top while dense baselines decay steeply and Qwen3.5 decays gently."
  caption="Soofi S pairs frontier-level capability with the highest measured aggregate long-context decode throughput, and unlike dense full-attention baselines holds it as context grows to 256K. Throughput is measured TP=1, one B200, batch 32, latency-subtraction; the Capability Index is an author-defined average of five benchmark groups, each normalized to the best plotted model (Soofi S Pretraining Report v1.0, Figure 1)."
/>

## The architecture: hybrid backbone, sparse experts

Soofi S reuses the openly published **Nemotron 3 Nano** reference design without modification — a
deliberate choice, so the effect of the German–English data recipe can be measured against an
architecture-identical control (the same-arch [Nemotron](/articles/nemotron-nvfp4) baseline). The
backbone is 52 layers: **23 Mamba-2** sequence-mixing layers, **23 granular MoE** layers, and only **6
Grouped-Query Attention** layers, distributed sparsely through the depth. Scrub the stack — and note how
few layers actually hold a cache:

<HybridStack />

The Mamba-2 layers carry most of the sequence mixing with a *fixed-size recurrent state*; the attention
layers give exact long-range recall but are the only ones whose cache grows. The capacity lives in the
MoE layers, and this is where "30B at the cost of 3B" comes from. Each MoE layer has **128 routed
experts** plus **2 shared experts**; a learned, sigmoid-gated router activates just **6 routed experts**
per token, with the 2 shared always on:

<MoeRouting />

The exact config, for reproducibility (Table 1): model dimension 2688, 32 attention query heads over just
2 KV heads (head dim 128), Mamba-2 state dimension 128, expert dimension 1856, squared-ReLU MoE
activation, RMSNorm, no positional embeddings, untied embeddings. Total 31.6B parameters, ~3.2B active
per token (~3.6B including embeddings).

## Why the cache stays near-constant

Decoding is memory-bandwidth bound: every generated token must re-read the model weights **and** the
attention cache of every sequence in the batch. In a dense full-attention model that per-sequence cache
grows with context, so at tens or hundreds of thousands of tokens, served many-at-once, the KV reads come
to dominate and throughput decays. Soofi's hybrid backbone attacks exactly this. Only 6 of 52 layers keep
a KV cache, with 2 KV heads each, so the incremental attention-cache footprint is about **6 KB per token
per sequence** — the report puts that at **11–53× lower** than the dense models in its comparison. As
context grows, only that small attention component scales with length; the Mamba-2 recurrent state stays
constant-size.

The measured payoff: at 40K context and batch 32, Soofi sustains **4.82k aggregate decode TPS/GPU**, a
reported **9.2×** over Ministral 3 14B, while fitting the weights and all 32 sequence states on a single
GPU. Across 4K→256K the aggregate decode rate stays essentially flat (no point more than ~34% below the
4K value), where dense throughput decays with context. Among the comparison models only Qwen3.5 — itself
a Gated-DeltaNet hybrid — scales similarly, and its 35B-A3B variant still measures ~1.9× slower than Soofi
at 40K. The prefill side shows the same shape: time-to-first-token at 256K is 372.7s for Soofi versus
2,058.9s for dense Ministral 3 14B and 6,428.6s for a dense Qwen3 32B control.

## The data: ~27T tokens, German on purpose

Soofi S was pretrained on approximately **27 trillion tokens** (~26.68T actually consumed) under a
three-phase Warmup–Stable–Decay curriculum: ~20T of diverse, quality-tiered pretraining, ~6.58T of
high-quality annealing, and a ~0.10T long-context extension that pushes the usable window to 1M tokens.
The defining move is that **German is deliberately up-weighted** — to 7.2% of the stable phase and 15.32%
of the annealing mixture, more than triple the ~5% total multilingual share of the reference Nemotron
recipe, and concentrated in a single language rather than spread across dozens.

<Figure
  src="/articles/soofi-s/fig2.png"
  alt="A flow (Sankey) diagram tracing seven data categories — English Web, Academic & Wiki, SFT, Reasoning, Code, Math, German — across three training phases. Phase 1 (~23T effective tokens) is dominated by English Web at 50.3%; Phase 2 (~6T) shifts toward skill data and raises German to 15.3%; Phase 3 (~188B) branches SFT into General, Code, and Math SFT for long-context extension."
  caption="The effective-token mixture across the three phases. Phase 1 maximizes diversity (50.3% English web); Phase 2 concentrates skill-oriented data and triples German's share to 15.3%; Phase 3 is a length-bucketed long-context pool. Band width is each source's share of that phase's tokens (Soofi S Pretraining Report v1.0, Figure 3)."
/>

The corpus is documented at the granularity of individual source datasets — raw tokens, epoch multiplier,
effective tokens, and even sources that were evaluated and *excluded* — so the mixture can be audited and,
where licenses permit, rebuilt. German coverage combines naturally occurring web and document text (HPLT,
German Commons, Genios, German FinePDFs/FineWiki) with machine-translated and synthetic German, since
high-quality native German text is far scarcer than English. The report identifies that German data
pipeline as the principal bottleneck for further gains.

Just as notable is *where* it ran. Soofi S was trained end-to-end on the **Industrial AI Cloud** operated
by Deutsche Telekom in Munich — up to **512 NVIDIA B200 GPUs** (64 DGX B200 nodes), ~253,000 B200
GPU-hours from 24 March to 13 May 2026, on a facility powered by renewable energy and cooled with water
from the Eisbach canal. Training on German soil under European data-protection rules is itself part of the
"sovereign" claim.

## Results, and how to read them

Against a set of large open-source models (Alia 40B, EuroLLM 22B, Apertus 70B, Olmo 3 32B), Soofi S is
the strongest in the set: highest **German aggregate** and, among fully open models, the highest English
and German evaluation scores — ahead of Olmo 3 32B and Apertus 70B despite activating a fraction of their
parameters.

<BenchBars
  title="German aggregate — open-source comparison (%)"
  unit="%"
  bars={[
    { label: "Soofi S 30B-A3B", value: 79.1, highlight: true },
    { label: "Apertus 70B", value: 72.8 },
    { label: "EuroLLM 22B", value: 70.6 },
    { label: "Olmo 3 32B", value: 69.2 },
    { label: "Alia 40B", value: 68.4 },
  ]}
/>

The point the authors most want to land is capability-per-active-parameter: Soofi matches dense 14–27B
models on English and German aggregates while activating only ~3.2B parameters per token.

<BenchBars
  title="Active parameters per token (B) — Soofi vs the open-source baselines"
  unit="B"
  bars={[
    { label: "Apertus 70B", value: 70 },
    { label: "Olmo 3 32B", value: 32 },
    { label: "EuroLLM 22B", value: 22 },
    { label: "Alia 40B", value: 40 },
    { label: "Soofi S (active)", value: 3.2, highlight: true },
  ]}
/>

It also posts the best code aggregates in that comparison (HumanEval 73.8, MBPP 70.2, HumanEval-DE 65.5,
MBPP-DE 84.2 — first on four of five code benchmarks), and leads the set on mathematics (GSM8K 86.1).

<Figure
  src="/articles/soofi-s/fig3.png"
  alt="A grouped bar chart, 'Base model evaluation overview', comparing Soofi S 30B-A3B (highlighted) against Alia 40B, EuroLLM 22B, Apertus 70B and Olmo 3 32B across English aggregate, German aggregate, Code EN, Code DE, Math EN, Math DE, MMLU-Pro and GPQA-D-DE. Soofi S is marked #1 on every group shown, with values 70.1, 79.1, 72.0, 74.9, 82.8, 71.5, 51.4 and 41.9."
  caption="Base-model evaluation overview for the open-source comparison: Soofi S is #1 on every aggregate shown against Alia, EuroLLM, Apertus and Olmo 3. Aggregates are harness-level suite means; results are author-reported on the authors' selected suite (Soofi S Pretraining Report v1.0, Figure 4)."
/>

Read against the *open-weight* set, the picture is more measured and the report says so: Soofi is **not**
the top model on aggregate — Qwen3.5 35B-A3B leads (English 74.6, German 81.6), and Soofi's English
aggregate (70.1) essentially ties Gemma 3 27B and Ministral 3 14B (both 70.3). Its clearest, cleanest
result is the architecture-identical comparison: versus Nemotron 3 Nano 30B-A3B (same backbone, different
data), the German–English recipe lifts German aggregate +4.2, held-out English +6.7, GPQA-Diamond +9.6,
and German-language proficiency (GLP-DE) +15.1 — while English capability is preserved or improved, the
usual price of monolingual specialization avoided.

## The honest caveats

<Callout type="warn">
  This is a **consortium tech report, not a peer-reviewed paper**. Every number is **author-reported**,
  the "Capability Index" in Figure 1 is **author-defined** (an average of five benchmark groups, each
  normalized to the best-plotted model), and the throughput figures use an **author-selected baseline set
  and measurement protocol** (TP=1, one B200, batch 32, latency-subtraction). Soofi S is a **3B-active**
  model: "matches dense 14–27B" is active-vs-total, not 30B-dense compute. "Best/highest among fully open"
  and "outperforms every European sovereign baseline" are **scoped to their comparison and eval suite**,
  and deliberate German up-weighting shapes the aggregates — so these are not unqualified SOTA claims.
</Callout>

The report is candid about its own limitations. **Competition-style math in German** is the clearest gap
to the frontier: Minerva MATH-DE 56.0 trails Qwen3.5 35B-A3B (76.5) and Gemma 3 27B (65.6). **Open-domain
factual recall** is capacity-limited — NaturalQuestions 79.0 trails the largest dense baselines (Gemma 3
27B 83.5), consistent with storing world knowledge in only ~3B active parameters (the authors expect
retrieval-augmentation to close this in practice). And on openness itself the report draws an explicit
line: it satisfies the OSI's OSAID 1.0 (weights, checkpoints, training and eval code, exact per-source
data accounting, all under permissive licenses), but falls short of the stricter "every training token
must be redistributable" bar on exactly one component — the commercially licensed Genios corpus (1.3% of
Phase 1) — so ~99% of the mixture, not 100%, can be independently reconstructed.

## The take

Soofi S's contribution is less a new mechanism than a **thesis about deployment cost, executed
transparently**. The near-constant cache is the load-bearing idea: by keeping only 6 of 52 layers as
attention and letting Mamba-2 carry the rest with a fixed-size state, decode throughput stops caring about
context length — which is where dense models bleed. Wrap that in a sparse MoE (3.2B active of 31.6B) and
you get a model that serves like a 3B but scores like a 14–27B dense on its target languages. Set the
knobs honestly — author-reported numbers, an author-defined capability index, a scoped baseline set, a 3B
active budget, real gaps in German competition math and factual recall — and what remains is genuinely
notable: a fully documented, per-source-audited, German–English pretraining run on sovereign European
B200 hardware, released with checkpoints and code. As a template for "open in substance, efficient by
architecture, and built at home," it is a clean and unusually legible bet.

---

*Built on the [Soofi S Pretraining Report v1.0](https://huggingface.co/Soofi-Project) ("A Sovereign,
Open-Source Foundation Model for German and English", the Soofi-Team; consortium coordinated by the KI
Bundesverband, funded by the German BMWE). Architecture, data, and benchmark figures are quoted from the
report for commentary; the interactive diagrams are illustrations of the mechanism, and the throughput
curves use the report's measured endpoints with an illustrative in-between shape. Related: the
architecture-shared [Nemotron in NVFP4](/articles/nemotron-nvfp4), [mixture-of-experts from
scratch](/articles/mixture-of-experts-from-scratch), [how LLM inference
works](/articles/how-llm-inference-works), and [large-scale
pretraining](/articles/megatrain-single-gpu-training).*