Monolith 1.0: a 1.6T open MoE built for reasoning

Basalt Labs' Monolith 1.0 is a 1.572-trillion-parameter Mixture-of-Experts with 49.5B active per token, released as open weights under an MIT license — weights, tokenizer, and eval harness, commercial use allowed. It is a decoder-only, reasoning-focused, Chinese-English model, and Basalt is blunt about what it is for: "a 1.6T open Mixture-of-Experts foundation model for reasoning at scale." The tech report lays out the recipe.

The headline number is the total parameter count, but the number that governs everything else is the active one. Monolith spends 1.57T parameters of capacity but pays for only 49.5B of compute per token — a 32x sparsity ratio. This piece walks the pieces that make that work: the MoE routing, the two-stage context extension to a million tokens, the training recipe, and the decode trick that makes a model this large servable. Full prose and math carry each idea; the interactive diagrams are there to build intuition.

The spec sheet

• 1.572T total parameters, 49.5B active per token — a 32x sparsity ratio.
• 80 layers, model dimension 8,192.
• Grouped-query attention: 64 query heads, 8 KV groups, head dim 128.
• 128 routed experts per layer (SwiGLU, intermediate dim 6,144) + 1 shared expert; top-2 routing.
• RoPE base 5e6, two-stage YaRN; context length 1,048,576 tokens.
• Byte-level BPE tokenizer, 151,936 vocab.

Two design choices do most of the work: how the experts are routed, and how the context is stretched. Take them one at a time.

Routing: top-2 of 128, plus one that never sleeps

Monolith's feed-forward block is a mixture of experts. Each layer holds 128 routed experts and one shared expert. For every token, a router scores the 128 and keeps the top-2; the shared expert is always on. So 3 of 129 experts fire per token. Scrub the token index and watch the routed pair swing while the shared expert stays lit:

The always-on shared expert is the part worth dwelling on. A pure top-$k$ router has to relearn the common, every-token computation inside many different experts, which wastes capacity. Splitting off one shared expert lets the routed experts specialize while the shared one carries the baseline work — a pattern that has become standard in large MoEs because it stabilizes routing at high sparsity.

The sparsity is what makes the size affordable. Active parameters are the shared expert plus the top-2 routed, so the per-token compute is that of a roughly 50B model while the knowledge capacity is that of a 1.57T one:

That 32x is the whole bet: you get the memory footprint of a trillion-scale model but the FLOPs of a mid-size one, provided you can route well and keep every expert busy.

Attention: grouped-query, so the 1M cache fits

Before the context trick, one attention detail matters for it. Monolith uses grouped-query attention — 64 query heads sharing just 8 KV groups. The KV cache stores keys and values per group, not per query head, so it is 8x smaller than full multi-head attention at the same width. At a million-token context the KV cache is the dominant memory cost of decoding, so an 8x reduction there is the difference between a 1M window being a spec and being something you can actually hold in memory.

Long context: two YaRN stages to a million tokens

Monolith pretrains at a cheap 4,096-token window, then extends to 1,048,576 tokens — a 256x stretch — using YaRN in two stages on a RoPE base of 5e6. YaRN rescales the rotary position frequencies so positions far beyond the training length stay in distribution instead of aliasing into nonsense. Doing it in two 16x steps rather than one 256x leap keeps long-range attention coherent. Step through the stages:

The extension factor is exactly

and a two-stage split puts the midpoint at the geometric mean, $\sqrt{256} = 16$, so each stage is a 16x reach: $4096 \to 65536 \to 1048576$. (The 65,536 midpoint is my illustration of a clean two-stage split; Basalt reports two YaRN stages without pinning the intermediate length.) The reason to stage it is numerical: RoPE extrapolation degrades faster than linearly with the extension factor, so two moderate stretches with a re-anchor in between hold up where one aggressive stretch smears the attention over distant tokens.

Training: 60T tokens, and a FLOP budget that checks out

Monolith is trained on 60T tokens (a multilingual mixture) in BF16 mixed precision with an FP32 optimizer state. Basalt reports a compute budget of about 1.8e25 FLOPs. That number is not arbitrary — the standard estimate for transformer training compute is

with $N_{\text{active}}$ the active parameters and $D$ the token count. MoE training compute scales with the active parameters, not the total, because only the active experts run per token. Plug in $N_{\text{active}} = 49.5\text{B}$ and $D = 60\text{T}$:

which lands on the reported 1.8e25. The sparsity pays off twice: at 32x it means a 1.57T model trains at the per-token cost of a ~50B one.

Post-training is a three-stage reasoning pipeline: SFT, then DPO, then RLVR — reinforcement learning with verifiable rewards. RLVR is the reasoning-specific piece: instead of a learned reward model, the reward comes from checking whether the answer is actually correct (a math result that verifies, code that passes tests), which is a cleaner signal for training long chains of thought and harder to reward-hack than a preference model.

Serving it: self-speculative decoding

A 1.57T-parameter model is memory-bound at decode time, so Monolith ships self-speculative decoding: the model drafts several tokens cheaply, then verifies them all in one forward pass, keeping the longest correct prefix and correcting the first miss. Scrub the phase and flip the domain:

Because the verify pass reproduces the base model exactly, this is lossless — the output is token for token what greedy decoding would have produced, just fewer expensive passes to get there. Code is more predictable than prose, so more drafts survive verification: Basalt reports ~2.1x faster decoding on natural language and ~2.7x on code.

Even so, "open weights" here still means rack-scale hardware. Basalt targets one GB300 NVL72 rack (72 GPUs) at FP8, or a CloudMatrix-384 at native BF16. You can download the weights; running them is another matter.

The benchmarks

Here is where honesty has to lead. On Basalt's own harness, at maximum thinking effort, Monolith posts numbers that are not just ahead of the field but near the ceiling of the tests themselves. Compared against GPT-5.4, Claude Opus 4.6, Gemini 3.1 Pro, and Kimi K2.6:

Read those bars, then read the next box before you form an opinion.

What I make of it

• The engineering is coherent and legible. 32x sparsity via top-2-of-128 plus a shared expert, an 8x-smaller KV cache from grouped-query attention, a staged YaRN reach to 1M tokens, and a FLOP budget that checks out against $6 N_{\text{active}} D$ — none of it is exotic, all of it is the right lever for a trillion-scale reasoning model. The self-speculative decode is a real, lossless serving win.
• The MIT license is the genuinely useful part. Weights, tokenizer, and eval harness, commercial use allowed — that is more open than most models at this scale, and it means the benchmark claims can, in principle, be checked by anyone with the hardware.
• The benchmarks are the part to hold at arm's length. Saturated, self-reported, own-harness scores are a marketing artifact until someone independent reproduces them. I would love to be wrong; I would rather wait for the third-party numbers than quote 99.4 as if it were settled.

The bet Monolith makes is that a trillion-scale open MoE, routed and staged carefully, can be a frontier reasoning model in the open. The architecture is a credible version of that bet. Whether it actually reasons at 99.4-on-HLE levels is a question its own harness cannot answer.

Sources: the Monolith 1.0 model card and the Basalt Labs tech report (architecture, training, deployment, benchmarks). Benchmark numbers are quoted as reported by Basalt on their own harness at maximum thinking effort. The training-compute figure is checked against $C \approx 6\,N_{\text{active}}\,D$ with the reported 49.5B active parameters and 60T tokens. The interactive diagrams illustrate the mechanisms; the routing, context, and decode visuals are illustrative, and the 65,536-token YaRN midpoint is my own clean two-stage split, not a disclosed intermediate length.