# Laguna's Model Factory: treating model development as an industrial process

> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/laguna-model-factory
> date: 2026-07-03
> tags: llm, agents, mixture-of-experts, systems, explainer
Most model reports are about a model. Poolside's **Laguna** report is unusual: its real subject
is the *factory*. Yes, it ships two Mixture-of-Experts models for agentic software engineering —
**Laguna M.1** (225.8B total, 23.4B active) and the open **Laguna XS.2** (33.4B total, 3B active) —
and they're solid, competitive-in-class coding models. But the argument the report actually makes
is that the models are downstream of something more valuable: a **Model Factory** that treats
foundation-model development as an *industrial process* rather than a craft. The evidence they
offer is a number — they built XS.2 from inception to release in **five weeks**.

<ModelFactory />

## The two models, briefly

The models are conventional-but-careful MoE. Both are pre-norm Transformers with RMSNorm and
token-choice routing with a shared expert. The report spells out XS.2's config — **8 of 256
routed experts** per token plus one shared expert, the routed output scaled by 2.5 before it
rejoins the residual (a DeepSeek-V3 / Nemotron-style modulation), a sigmoid router normalized
after top-k, and a dense first layer for stability — but gives M.1's totals without its routing
width, so I won't guess it. Both use GQA (not MLA), the **Muon** optimizer, and a served context
of **256K** — XS.2's attention runs 8 KV heads with softplus per-head gating.

<TwoModels />

The one architectural wrinkle worth noting is XS.2's attention: where M.1 runs **global attention
on every layer**, XS.2 interleaves **sliding-window and global at a 3:1 ratio** with a 512-token
window — the same [hybrid-attention idea as MiMo-V2-Flash](/articles/mimo-v2-flash), tuned for a
smaller model that has to be cheap to serve. And the data is unambiguously code-first: XS.2's
pretraining mix is ~30% raw code plus a large synthetic-code share, on top of >30T tokens (from a
pool of ~27T unique). But by the report's own framing, none of this is the point. The point is how
it was all *made*.

XS.2 is really M.1 with four ablated deltas flipped on: the hybrid attention above, a
Warmup-Stable-Decay LR schedule instead of cosine, the routed-expert modulation, and one dense
layer instead of three. Each was a config change against the M.1 baseline, chosen on a small MoE
proxy — cheap precisely because the factory made the data pipeline, trainer, and eval harness
transfer for free. The peak LR came from a fitted **WSD scaling law**, $\text{lr}^\star \propto
N^{-0.46}\,D^{-0.27}$ in active params and tokens, rather than a re-tuned sweep. The stability
lessons transferred too: M.1's pre-training surfaced expert collapse around **450B tokens** (fixed
by Moonlight-style LR scaling, so Muon runs at AdamW-scale weight decay) and a logit-drift blowup
traced to a BF16 all-reduce on the LM-head input gradient (fixed by forcing that one reduction to
FP32). XS.2 hit none of them — the factory's job is to make the second model boring.

## What "industrial process" actually means

The Model Factory is defined as "a tightly-integrated stack of versioned data, training, evaluation,
and inference." Three principles hold it together:

- **Experiments as code.** Every run's inputs and config are committed to one repo and get a unique
  ID; a Dagster DAG is the control plane for what runs and what depends on what. The payoff is
  *end-to-end lineage* — a single token in a packed training shard traces back through dedup,
  filtering, and synthesis to its source document, and every checkpoint, eval, and deployment traces
  to the exact run that produced it. Nothing is a mystery artifact.
- **One code base for research and production.** A promising research idea isn't re-implemented to
  ship — it's "promoted into production by flipping a configuration flag." The inference library
  (Atlas, on vLLM) consumes the trainer's (Titan) model definitions *bit-accurately*, so what you
  evaluate is what you serve is what generates your RL rollouts.
- **Reserve human attention for novel decisions.** A custom Kubernetes scheduler places jobs in
  under a minute, auto-recovers from hardware failure, and only pages a human when recovery itself
  fails. Cross-replica bit-identical weight-hash checks catch the silent data corruption a defective
  GPU would otherwise smear through a run.

The components have names, and the flywheel is literal: **Titan** trains, **Blender** streams the
data mix, **Hive** generates synthetic data, **Atlas** serves inference *and* RL rollouts, and a
containerized **Code Execution** platform spanning ~1M repositories provides synthetic tasks,
evaluation, *and* RL execution rewards — one component doing all three. The RL harness they train
against is the same harness they ship to customers. That's the loop the interactive above is
gesturing at: data → train → eval → infer, where inference feeds the next round of data, and the
whole thing is versioned tightly enough that a five-week model is possible.

<Figure
  src="/articles/laguna-model-factory/fig1.png"
  alt="Left-to-right pipeline of labeled stages: Common Crawl, parsing, language ID, deduplication, quality tagging, conservative filtering, score-and-rank, bucketing, sampling, and final web mix."
  caption="One arm of the factory made concrete: the versioned web-data pipeline runs raw Common Crawl through extraction, dedup, quality tagging, conservative filtering, composite scoring, and quota-aware sampling into the training mix — an assembly line with end-to-end lineage (paper, Figure 3)."
/>

That pipeline also flipped a habit. For M.1 they ran a high-precision filter that aggressively
dropped noisy documents; for XS.2 they went **high-recall** instead — the composite score fully
rejects only ~25.8% of web samples as pure noise and *recovers ~34%* of documents the old static
rules had thrown away, then treats quality as a ranking signal and samples from score buckets
rather than hard-filtering. Under a >30T-token budget, controlling repetition and diversity beats
maximizing average quality; over-filtering starves the long tail.

## Choosing the data mix by optimization, not taste

The web pipeline decides *which documents survive*. A separate problem is *how much of each
source to train on* — the mixture weights. Done by hand, that's a few knobs set by taste and a
handful of ablations. The report's quietest radical move, **AutoMixer**, turns it into an
optimization loop, and it's the cleanest single instance of the factory thesis: a decision that
used to be craft becomes a versioned, automated search.

The setup is a surrogate-model sweep. For each data ablation they train a **swarm of ~60 proxy
models** — each a ~0.5B-parameter MoE on ~60B tokens — from **different mixtures** sampled over
**50+ heterogeneous dataset groups** (web, curated edu, academic, raw / grounded / synthetic code,
math web, conversational and knowledge sets). Every proxy is one labeled example of "mixture in,
capabilities out."

<Figure
  src="/articles/laguna-model-factory/fig4.png"
  alt="AutoMixer pipeline: a column of dataset groups on the left feeds a six-step loop — sample N mixtures over the simplex, train N proxy models, evaluate M capabilities, fit one regressor per capability, then optimize the mixture weights to maximize all capabilities jointly."
  caption="The AutoMixer loop: sample mixtures over the dataset-group simplex, train a proxy model on each, evaluate a small set of capabilities, fit a surrogate regressor per capability, then optimize the mixture. The diagram illustrates the loop with 15 example groups; the real sweep spans 50+ (paper, Figure 6)."
/>

Formally they learn a surrogate $\mathcal{M}: x \to y$ where $x \in \Delta^{d}$ is a mixture over
$d$ dataset groups and $y \in \mathbb{R}^{k}$ is a vector of downstream metrics across $k$
capability groups — coding, math reasoning, STEM knowledge, commonsense, general knowledge.
Candidate mixtures are drawn near a hand-designed prior $x_0$ as $x \sim \text{Dirichlet}(\alpha
x_0)$ subject to $\lVert x - x_0 \rVert_1 < \epsilon$, so the search stays in realistic regions.
For each capability $j$ they fit a regressor $f_j(x) \approx y_j$ — linear in the simplified
picture, $\hat{y}_j = \beta_j^{\top} x + b_j$, non-linear in practice. The mixture is then chosen
by maximizing a weighted sum of the surrogates over the simplex:

$$
\max_{x}\ \sum_{j=1}^{k} w_j\, f_j(x)
\quad \text{s.t.}\quad \sum_i x_i = 1,\ \ x_i \ge 0,\ \ \lVert x - x_0 \rVert_1 < \epsilon
$$

with a $\lambda\, D_{\mathrm{KL}}(x \Vert x_0)$ penalty keeping the answer from collapsing onto a
few dominant sources. The knobs $w_j$ are where intent enters: weight coding and math and the
optimizer allocates data toward them.

<AutoMixer />

The learned surrogate recovers relationships you'd expect — synthetic and curated code lift coding
evals; conversational and knowledge corpora lift commonsense — plus finer cross-effects. On a
3B-param / 1.5T-token check, the optimized mix posts large gains on the targets (HumanEval+ **+43%**,
CRUX-I **+54%**, GSM8K **+41%**, MultiPL-E **+27%**) and, encouragingly, **generalizes to held-out
benchmarks** it wasn't optimized against (MATH **+25%**, LiveCodeBench **+39%**, BigCodeBench +16%).
The cost is stated honestly and it's small: a few commonsense tasks regress (ARC-C **−6.8%**, the
rest under 1.5%), which is exactly what you sign up for when the objective down-weights them. XS.2's
final mixture — **30.6% raw code, 25.4% synthetic/code-text, 25.2% web, 9% math**, the rest
knowledge / instruction / academic / books (Table 4) — shifted toward web, synthetic, and math
relative to M.1's while keeping the code-heavy spine. That's the thesis in one artifact: a data-mix
decision made by an optimizer over a learned model, logged and re-runnable, instead of argued in a
meeting.

## Agentic training, from real commits

The coding ability comes from training on the actual job. Poolside turns **real git commits from
public repos into verifiable tasks** — a problem statement, a repo checkout, and a hidden test
patch, with the gold answer being the commit's own diff. A two-sided filter keeps only commits
where the gold diff passes the tests *and* an empty patch fails them (discarding trivial or
non-exercising tests), yielding **30–60k tasks from a ~236k-commit pool**. Those tasks feed both
SFT (as teacher-generated trajectories, sometimes wrapped in synthetic system messages for
instruction-following pressure) and the RL pool (where the repo's own test suite is the verifier).

The RL stage is online: the policy itself drives the **production agent harness** across several
thousand live containers at a time, and each rollout's reward comes from that container's verifier.
The objective is **CISPO** — the importance-ratio-clipping surrogate from MiniMax-M1, not a Poolside
invention — paired with a length-weighted leave-one-out group baseline; clipping is asymmetric,
an effective $[0, 5]$ on the ratio, so it only bites on heavily off-policy tokens. Reward is a
deterministic chain of checks: a malformed tool call or template violation is $-0.1$, giving up
before a minimum number of tool calls is $-0.1$, a timeout is $0.0$, and the **only positive reward
is the binary task verifier** ($1.0$) — unit tests for SWE tasks, bash assertions for terminal
tasks, exact-match for tool-integrated math. A small $-0.05$ per-token penalty lands on exactly the
tokens of a failing tool step to sharpen credit assignment; everything else is carried by the
terminal 1/0. It's the same "make the process the trainable target" instinct as [Agents-A1's
verifier-graded trajectories](/articles/agents-a1), wired straight into the factory — the execution
environment that grades RL is the one that generates data and runs evals.

## The numbers, honestly

Here's where the modest framing matters. The report claims the models are "competitive with
state-of-the-art open models in their respective weight classes," and that's accurate — *competitive*,
not leading. M.1 lands mid-pack among the ~200B-class open models on SWE-bench Verified:

<BenchBars
  title="SWE-bench Verified — ~200B-class open models (%)"
  unit="%"
  bars={[
    { label: "DeepSeek-V4-Flash", value: 79.0 },
    { label: "Qwen3.5 (397B-A17B)", value: 76.2 },
    { label: "Laguna M.1 (225B-A23B)", value: 74.6, highlight: true },
    { label: "GLM-4.7 (355B-A32B)", value: 73.8 },
    { label: "Devstral 2 (123B)", value: 72.2 },
  ]}
/>

<Figure
  src="/articles/laguna-model-factory/fig2.png"
  alt="Grouped bar charts of Laguna M.1 versus Devstral 2, GLM-4.7, DeepSeek-V4-Flash, Qwen3.5 and Claude Sonnet 4.6 across four benchmarks: SWE-bench Verified, Multilingual, Pro, and Terminal-Bench 2.0."
  caption="The paper's own headline chart for M.1, across all four agentic benchmarks — not just SWE-bench Verified — versus ~200B-class open and frontier references (paper, Figure 1a)."
/>

XS.2 is the more interesting result, because it's competitive in the ~30B class while activating only
**3B** parameters per token — against dense 24–31B rivals:

<BenchBars
  title="SWE-bench Verified — ~30B-class open models (%)"
  unit="%"
  bars={[
    { label: "Qwen3.6 (35B-A3B)", value: 73.4 },
    { label: "Laguna XS.2 (33B-A3B)", value: 69.9, highlight: true },
    { label: "Qwen3.5 (35B-A3B)", value: 69.2 },
    { label: "Devstral Small 2 (24B)", value: 68.0 },
    { label: "Gemma 4 (31B)", value: 52.0 },
  ]}
/>

<Figure
  src="/articles/laguna-model-factory/fig3.png"
  alt="Grouped bar charts of Laguna XS.2 versus Devstral Small 2, Gemma 4, Qwen3.5, Qwen3.6, Claude Haiku 4.5 and GPT-5.4 Nano across SWE-bench Verified, Multilingual, Pro, and Terminal-Bench 2.0."
  caption="The same headline chart for XS.2 — competitive across all four benchmarks against ~30B-class open and frontier references while activating only 3B parameters per token (paper, Figure 1b)."
/>

XS.2 beats Devstral Small 2 and Gemma 4 and edges Qwen3.5, though Qwen3.6 leads the class. Two honesty
notes the report itself makes: the baseline numbers are Poolside-selected published scores (not re-run
in their harness, so there's provider-config bias), and they patched all four benchmarks to remove
git-history leaks before scoring — so these differ slightly from public leaderboards by construction.
It's the rare case where the *methodology* disclosure is more reassuring than the raw scores.

One finding from the quantization work is worth carrying away regardless of the leaderboard: **bad
quantization hurts agentic benchmarks far more than single-turn ones.** A small per-token error
that's invisible on a one-shot question compounds across a hundred-step trajectory, and the report
saw exactly that — intermediate schemes that barely moved single-turn scores cratered the agentic
ones. The fix started by looking at *where* the error comes from. Naive INT4 (AWQ, `W4A16`) lost
quality because outlier activations pile up in the residual stream starting around **layer 30 of
the 40-layer network**:

<Figure
  src="/articles/laguna-model-factory/fig5.png"
  alt="Line chart of residual-stream activation magnitude by layer: the median stays near zero across all 40 layers while the per-layer maximum stays small until about layer 30, then jumps to roughly 90 and stays high through the last layers."
  caption="Residual-stream activations by layer for XS.2: the median is flat near zero, but the maximum explodes after layer 30 — the outliers that make a flat low-bit scheme unsafe for the late layers (paper, Figure 7)."
/>

So they went mixed-precision: **`INT4` for the first 30 layers, `INT8` for the last 10** (with a
SpinQuant rotation as a pre-pass). `NVFP4` needed more — direct post-training quantization lost too
much, so they recovered it with **quantization-aware distillation**, training the quantized student
to match a higher-precision teacher on a fixed dataset. The KV cache goes to `FP8` across the full
131K context, roughly doubling how many trajectories a replica can hold. The through-line is the
same as everything else here: the diverse eval harness is what caught the agentic-only regression
that a single-turn benchmark would have waved through.

## The take

I went in expecting an architecture paper and came out thinking about CI. The Laguna models are
genuinely fine — a competent ~200B flagship and a genuinely efficient 3B-active open model that holds
its own in a crowded class — but they're not what the report is selling. It's selling the claim that
*iteration speed* is the frontier lever: if every run is reproducible code, every win ships by flipping
a flag, and the same execution environment grades your RL, runs your evals, and generates your data,
then you can turn out a from-scratch model in five weeks and keep pace with model complexity instead of
drowning in it. Whether the "factory is the moat" thesis holds as everyone industrializes is the open
question — but as a piece of honest infrastructure writing, with the models presented as *outputs of a
process* rather than heroic artifacts, it's a refreshing shape for a technical report. XS.2's weights
are open (Apache-2.0); the factory, of course, is not.

---

*Built on the [Laguna M.1/XS.2 Technical Report](https://arxiv.org/abs/2605.27605) (Poolside, 2026) and
the [XS.2 model release](https://huggingface.co/collections/poolside/laguna-xs2) (Apache-2.0). Benchmark
figures are from the report's tables (baselines are Poolside-selected published scores on leak-patched
benchmark images); SWE-bench Verified figures use the report/model-card values (74.6 / 69.9), higher
than the earlier launch checkpoint.*