Laguna's Model Factory: treating model development as an industrial process

Most model reports are about a model. Poolside's Laguna report is unusual: its real subject is the factory. Yes, it ships two Mixture-of-Experts models for agentic software engineering — Laguna M.1 (225.8B total, 23.4B active) and the open Laguna XS.2 (33.4B total, 3B active) — and they're solid, competitive-in-class coding models. But the argument the report actually makes is that the models are downstream of something more valuable: a Model Factory that treats foundation-model development as an industrial process rather than a craft. The evidence they offer is a number — they built XS.2 from inception to release in five weeks.

The two models, briefly

The models are conventional-but-careful MoE. Both are pre-norm Transformers with RMSNorm and token-choice routing with a shared expert. The report spells out XS.2's config — 8 of 256 routed experts per token plus one shared expert, the routed output scaled by 2.5 before it rejoins the residual (a DeepSeek-V3 / Nemotron-style modulation), a sigmoid router normalized after top-k, and a dense first layer for stability — but gives M.1's totals without its routing width, so I won't guess it. Both use GQA (not MLA), the Muon optimizer, and a served context of 256K — XS.2's attention runs 8 KV heads with softplus per-head gating.

The one architectural wrinkle worth noting is XS.2's attention: where M.1 runs global attention on every layer, XS.2 interleaves sliding-window and global at a 3:1 ratio with a 512-token window — the same hybrid-attention idea as MiMo-V2-Flash, tuned for a smaller model that has to be cheap to serve. And the data is unambiguously code-first: XS.2's pretraining mix is ~30% raw code plus a large synthetic-code share, on top of >30T tokens (from a pool of ~27T unique). But by the report's own framing, none of this is the point. The point is how it was all made.

XS.2 is really M.1 with four ablated deltas flipped on: the hybrid attention above, a Warmup-Stable-Decay LR schedule instead of cosine, the routed-expert modulation, and one dense layer instead of three. Each was a config change against the M.1 baseline, chosen on a small MoE proxy — cheap precisely because the factory made the data pipeline, trainer, and eval harness transfer for free. The peak LR came from a fitted WSD scaling law, $\text{lr}^\star \propto N^{-0.46}\,D^{-0.27}$ in active params and tokens, rather than a re-tuned sweep. The stability lessons transferred too: M.1's pre-training surfaced expert collapse around 450B tokens (fixed by Moonlight-style LR scaling, so Muon runs at AdamW-scale weight decay) and a logit-drift blowup traced to a BF16 all-reduce on the LM-head input gradient (fixed by forcing that one reduction to FP32). XS.2 hit none of them — the factory's job is to make the second model boring.

What "industrial process" actually means

The Model Factory is defined as "a tightly-integrated stack of versioned data, training, evaluation, and inference." Three principles hold it together:

• Experiments as code. Every run's inputs and config are committed to one repo and get a unique ID; a Dagster DAG is the control plane for what runs and what depends on what. The payoff is end-to-end lineage — a single token in a packed training shard traces back through dedup, filtering, and synthesis to its source document, and every checkpoint, eval, and deployment traces to the exact run that produced it. Nothing is a mystery artifact.
• One code base for research and production. A promising research idea isn't re-implemented to ship — it's "promoted into production by flipping a configuration flag." The inference library (Atlas, on vLLM) consumes the trainer's (Titan) model definitions bit-accurately, so what you evaluate is what you serve is what generates your RL rollouts.
• Reserve human attention for novel decisions. A custom Kubernetes scheduler places jobs in under a minute, auto-recovers from hardware failure, and only pages a human when recovery itself fails. Cross-replica bit-identical weight-hash checks catch the silent data corruption a defective GPU would otherwise smear through a run.

The components have names, and the flywheel is literal: Titan trains, Blender streams the data mix, Hive generates synthetic data, Atlas serves inference and RL rollouts, and a containerized Code Execution platform spanning ~1M repositories provides synthetic tasks, evaluation, and RL execution rewards — one component doing all three. The RL harness they train against is the same harness they ship to customers. That's the loop the interactive above is gesturing at: data → train → eval → infer, where inference feeds the next round of data, and the whole thing is versioned tightly enough that a five-week model is possible.

expand

That pipeline also flipped a habit. For M.1 they ran a high-precision filter that aggressively dropped noisy documents; for XS.2 they went high-recall instead — the composite score fully rejects only ~25.8% of web samples as pure noise and recovers ~34% of documents the old static rules had thrown away, then treats quality as a ranking signal and samples from score buckets rather than hard-filtering. Under a >30T-token budget, controlling repetition and diversity beats maximizing average quality; over-filtering starves the long tail.

Choosing the data mix by optimization, not taste

The web pipeline decides which documents survive. A separate problem is how much of each source to train on — the mixture weights. Done by hand, that's a few knobs set by taste and a handful of ablations. The report's quietest radical move, AutoMixer, turns it into an optimization loop, and it's the cleanest single instance of the factory thesis: a decision that used to be craft becomes a versioned, automated search.

The setup is a surrogate-model sweep. For each data ablation they train a swarm of ~60 proxy models — each a ~0.5B-parameter MoE on ~60B tokens — from different mixtures sampled over 50+ heterogeneous dataset groups (web, curated edu, academic, raw / grounded / synthetic code, math web, conversational and knowledge sets). Every proxy is one labeled example of "mixture in, capabilities out."

expand

Formally they learn a surrogate $\mathcal{M}: x \to y$ where $x \in \Delta^{d}$ is a mixture over $d$ dataset groups and $y \in \mathbb{R}^{k}$ is a vector of downstream metrics across $k$ capability groups — coding, math reasoning, STEM knowledge, commonsense, general knowledge. Candidate mixtures are drawn near a hand-designed prior $x_0$ as $x \sim \text{Dirichlet}(\alpha x_0)$ subject to $\lVert x - x_0 \rVert_1 < \epsilon$, so the search stays in realistic regions. For each capability $j$ they fit a regressor $f_j(x) \approx y_j$ — linear in the simplified picture, $\hat{y}_j = \beta_j^{\top} x + b_j$, non-linear in practice. The mixture is then chosen by maximizing a weighted sum of the surrogates over the simplex:

with a $\lambda\, D_{\mathrm{KL}}(x \Vert x_0)$ penalty keeping the answer from collapsing onto a few dominant sources. The knobs $w_j$ are where intent enters: weight coding and math and the optimizer allocates data toward them.

The learned surrogate recovers relationships you'd expect — synthetic and curated code lift coding evals; conversational and knowledge corpora lift commonsense — plus finer cross-effects. On a 3B-param / 1.5T-token check, the optimized mix posts large gains on the targets (HumanEval+ +43%, CRUX-I +54%, GSM8K +41%, MultiPL-E +27%) and, encouragingly, generalizes to held-out benchmarks it wasn't optimized against (MATH +25%, LiveCodeBench +39%, BigCodeBench +16%). The cost is stated honestly and it's small: a few commonsense tasks regress (ARC-C −6.8%, the rest under 1.5%), which is exactly what you sign up for when the objective down-weights them. XS.2's final mixture — 30.6% raw code, 25.4% synthetic/code-text, 25.2% web, 9% math, the rest knowledge / instruction / academic / books (Table 4) — shifted toward web, synthetic, and math relative to M.1's while keeping the code-heavy spine. That's the thesis in one artifact: a data-mix decision made by an optimizer over a learned model, logged and re-runnable, instead of argued in a meeting.

Agentic training, from real commits

The coding ability comes from training on the actual job. Poolside turns real git commits from public repos into verifiable tasks — a problem statement, a repo checkout, and a hidden test patch, with the gold answer being the commit's own diff. A two-sided filter keeps only commits where the gold diff passes the tests and an empty patch fails them (discarding trivial or non-exercising tests), yielding 30–60k tasks from a ~236k-commit pool. Those tasks feed both SFT (as teacher-generated trajectories, sometimes wrapped in synthetic system messages for instruction-following pressure) and the RL pool (where the repo's own test suite is the verifier).

The RL stage is online: the policy itself drives the production agent harness across several thousand live containers at a time, and each rollout's reward comes from that container's verifier. The objective is CISPO — the importance-ratio-clipping surrogate from MiniMax-M1, not a Poolside invention — paired with a length-weighted leave-one-out group baseline; clipping is asymmetric, an effective $[0, 5]$ on the ratio, so it only bites on heavily off-policy tokens. Reward is a deterministic chain of checks: a malformed tool call or template violation is $-0.1$, giving up before a minimum number of tool calls is $-0.1$, a timeout is $0.0$, and the only positive reward is the binary task verifier ($1.0$) — unit tests for SWE tasks, bash assertions for terminal tasks, exact-match for tool-integrated math. A small $-0.05$ per-token penalty lands on exactly the tokens of a failing tool step to sharpen credit assignment; everything else is carried by the terminal 1/0. It's the same "make the process the trainable target" instinct as Agents-A1's verifier-graded trajectories, wired straight into the factory — the execution environment that grades RL is the one that generates data and runs evals.

The numbers, honestly

Here's where the modest framing matters. The report claims the models are "competitive with state-of-the-art open models in their respective weight classes," and that's accurate — competitive, not leading. M.1 lands mid-pack among the ~200B-class open models on SWE-bench Verified:

expand

XS.2 is the more interesting result, because it's competitive in the ~30B class while activating only 3B parameters per token — against dense 24–31B rivals:

expand

XS.2 beats Devstral Small 2 and Gemma 4 and edges Qwen3.5, though Qwen3.6 leads the class. Two honesty notes the report itself makes: the baseline numbers are Poolside-selected published scores (not re-run in their harness, so there's provider-config bias), and they patched all four benchmarks to remove git-history leaks before scoring — so these differ slightly from public leaderboards by construction. It's the rare case where the methodology disclosure is more reassuring than the raw scores.

One finding from the quantization work is worth carrying away regardless of the leaderboard: bad quantization hurts agentic benchmarks far more than single-turn ones. A small per-token error that's invisible on a one-shot question compounds across a hundred-step trajectory, and the report saw exactly that — intermediate schemes that barely moved single-turn scores cratered the agentic ones. The fix started by looking at where the error comes from. Naive INT4 (AWQ, W4A16) lost quality because outlier activations pile up in the residual stream starting around layer 30 of the 40-layer network:

expand

So they went mixed-precision: INT4 for the first 30 layers, INT8 for the last 10 (with a SpinQuant rotation as a pre-pass). NVFP4 needed more — direct post-training quantization lost too much, so they recovered it with quantization-aware distillation, training the quantized student to match a higher-precision teacher on a fixed dataset. The KV cache goes to FP8 across the full 131K context, roughly doubling how many trajectories a replica can hold. The through-line is the same as everything else here: the diverse eval harness is what caught the agentic-only regression that a single-turn benchmark would have waved through.

The take

I went in expecting an architecture paper and came out thinking about CI. The Laguna models are genuinely fine — a competent ~200B flagship and a genuinely efficient 3B-active open model that holds its own in a crowded class — but they're not what the report is selling. It's selling the claim that iteration speed is the frontier lever: if every run is reproducible code, every win ships by flipping a flag, and the same execution environment grades your RL, runs your evals, and generates your data, then you can turn out a from-scratch model in five weeks and keep pace with model complexity instead of drowning in it. Whether the "factory is the moat" thesis holds as everyone industrializes is the open question — but as a piece of honest infrastructure writing, with the models presented as outputs of a process rather than heroic artifacts, it's a refreshing shape for a technical report. XS.2's weights are open (Apache-2.0); the factory, of course, is not.

Built on the Laguna M.1/XS.2 Technical Report (Poolside, 2026) and the XS.2 model release (Apache-2.0). Benchmark figures are from the report's tables (baselines are Poolside-selected published scores on leak-patched benchmark images); SWE-bench Verified figures use the report/model-card values (74.6 / 69.9), higher than the earlier launch checkpoint.