# Monolith 1.0: a 1.6T open MoE built for reasoning > Satyajit Ghana — Head of Engineering @ Inkers Technology > canonical: https://ai.thesatyajit.com/articles/monolith-1-0 > date: 2026-07-17 > tags: llm, mixture-of-experts, reasoning, long-context, explainer Basalt Labs' [Monolith 1.0](https://huggingface.co/basaltlabsai/monolith-1.0) is a **1.572-trillion-parameter** Mixture-of-Experts with **49.5B active per token**, released as open weights under an MIT license — weights, tokenizer, and eval harness, commercial use allowed. It is a decoder-only, reasoning-focused, Chinese-English model, and Basalt is blunt about what it is for: "a 1.6T open Mixture-of-Experts foundation model for reasoning at scale." The [tech report](https://basaltlabs.org/monolith) lays out the recipe. The headline number is the total parameter count, but the number that governs everything else is the **active** one. Monolith spends 1.57T parameters of capacity but pays for only 49.5B of compute per token — a **32x sparsity** ratio. This piece walks the pieces that make that work: the MoE routing, the two-stage context extension to a million tokens, the training recipe, and the decode trick that makes a model this large servable. Full prose and math carry each idea; the interactive diagrams are there to build intuition. ## The spec sheet - **1.572T** total parameters, **49.5B** active per token — a **32x** sparsity ratio. - **80 layers**, model dimension **8,192**. - **Grouped-query attention**: 64 query heads, 8 KV groups, head dim 128. - **128 routed experts** per layer (SwiGLU, intermediate dim 6,144) **+ 1 shared expert**; **top-2** routing. - **RoPE base 5e6**, two-stage YaRN; context length **1,048,576** tokens. - Byte-level BPE tokenizer, **151,936** vocab. Two design choices do most of the work: how the experts are routed, and how the context is stretched. Take them one at a time. ## Routing: top-2 of 128, plus one that never sleeps Monolith's feed-forward block is a mixture of experts. Each layer holds **128 routed experts** and **one shared expert**. For every token, a router scores the 128 and keeps the **top-2**; the shared expert is always on. So **3 of 129** experts fire per token. Scrub the token index and watch the routed pair swing while the shared expert stays lit: The always-on shared expert is the part worth dwelling on. A pure top-$k$ router has to relearn the common, every-token computation inside many different experts, which wastes capacity. Splitting off one shared expert lets the routed experts specialize while the shared one carries the baseline work — a pattern that has become standard in large MoEs because it stabilizes routing at high sparsity. The sparsity is what makes the size affordable. Active parameters are the shared expert plus the top-2 routed, so the per-token compute is that of a roughly 50B model while the knowledge capacity is that of a 1.57T one: $$ \text{sparsity} = \frac{N_{\text{total}}}{N_{\text{active}}} = \frac{1.572 \times 10^{12}}{4.95 \times 10^{10}} \approx 32\times $$ That 32x is the whole bet: you get the memory footprint of a trillion-scale model but the FLOPs of a mid-size one, provided you can route well and keep every expert busy. ## Attention: grouped-query, so the 1M cache fits Before the context trick, one attention detail matters for it. Monolith uses **grouped-query attention** — 64 query heads sharing just **8 KV groups**. The KV cache stores keys and values per group, not per query head, so it is **8x smaller** than full multi-head attention at the same width. At a million-token context the KV cache is the dominant memory cost of decoding, so an 8x reduction there is the difference between a 1M window being a spec and being something you can actually hold in memory. ## Long context: two YaRN stages to a million tokens Monolith pretrains at a cheap **4,096-token** window, then extends to **1,048,576** tokens — a **256x** stretch — using **YaRN** in two stages on a RoPE base of **5e6**. YaRN rescales the rotary position frequencies so positions far beyond the training length stay in distribution instead of aliasing into nonsense. Doing it in two 16x steps rather than one 256x leap keeps long-range attention coherent. Step through the stages: The extension factor is exactly $$ \frac{1048576}{4096} = 256, $$ and a two-stage split puts the midpoint at the geometric mean, $\sqrt{256} = 16$, so each stage is a 16x reach: $4096 \to 65536 \to 1048576$. (The 65,536 midpoint is my illustration of a clean two-stage split; Basalt reports two YaRN stages without pinning the intermediate length.) The reason to stage it is numerical: RoPE extrapolation degrades faster than linearly with the extension factor, so two moderate stretches with a re-anchor in between hold up where one aggressive stretch smears the attention over distant tokens. ## Training: 60T tokens, and a FLOP budget that checks out Monolith is trained on **60T tokens** (a multilingual mixture) in **BF16 mixed precision with an FP32 optimizer state**. Basalt reports a compute budget of about **1.8e25 FLOPs**. That number is not arbitrary — the standard estimate for transformer training compute is $$ C \approx 6 \, N_{\text{active}} \, D, $$ with $N_{\text{active}}$ the active parameters and $D$ the token count. MoE training compute scales with the **active** parameters, not the total, because only the active experts run per token. Plug in $N_{\text{active}} = 49.5\text{B}$ and $D = 60\text{T}$: $$ 6 \times (4.95 \times 10^{10}) \times (6.0 \times 10^{13}) \approx 1.78 \times 10^{25}\ \text{FLOPs}, $$ which lands on the reported 1.8e25. The sparsity pays off twice: at 32x it means a 1.57T model trains at the per-token cost of a ~50B one. Post-training is a three-stage reasoning pipeline: **SFT**, then **DPO**, then **RLVR** — reinforcement learning with verifiable rewards. RLVR is the reasoning-specific piece: instead of a learned reward model, the reward comes from checking whether the answer is actually correct (a math result that verifies, code that passes tests), which is a cleaner signal for training long chains of thought and harder to reward-hack than a preference model. ## Serving it: self-speculative decoding A 1.57T-parameter model is memory-bound at decode time, so Monolith ships **self-speculative decoding**: the model drafts several tokens cheaply, then verifies them all in one forward pass, keeping the longest correct prefix and correcting the first miss. Scrub the phase and flip the domain: Because the verify pass reproduces the base model exactly, this is **lossless** — the output is token for token what greedy decoding would have produced, just fewer expensive passes to get there. Code is more predictable than prose, so more drafts survive verification: Basalt reports **~2.1x** faster decoding on natural language and **~2.7x** on code. Even so, "open weights" here still means rack-scale hardware. Basalt targets **one GB300 NVL72 rack (72 GPUs)** at FP8, or a **CloudMatrix-384** at native BF16. You can download the weights; running them is another matter. ## The benchmarks Here is where honesty has to lead. On Basalt's own harness, at maximum thinking effort, Monolith posts numbers that are not just ahead of the field but near the ceiling of the tests themselves. Compared against GPT-5.4, Claude Opus 4.6, Gemini 3.1 Pro, and Kimi K2.6: Read those bars, then read the next box before you form an opinion. **These are single-lab, self-reported, own-harness numbers — treat them as directional, not settled.** A **99.4** on Humanity's Last Exam and a **100.0** on AIME 2025 are effectively saturation: the model is not beating the field by a few points, it is sitting at the top of the scale while every frontier model on the same chart trails by 30 to 40 points. Numbers that clean from the lab that built the model and ran the eval are exactly the ones to be skeptical of. Own-harness, maximum-effort results select for the configuration that flatters the model; they say nothing about a neutral setup, a contaminated-test check, or a different prompt. The thing to wait for is **independent third-party evaluation** on held-out sets. And the practical caveat does not go away with better scores: "open weights" for a 1.57T model still means a **GB300 NVL72 rack** to run it, so open here is a licensing fact, not an accessibility one. ## What I make of it - **The engineering is coherent and legible.** 32x sparsity via top-2-of-128 plus a shared expert, an 8x-smaller KV cache from grouped-query attention, a staged YaRN reach to 1M tokens, and a FLOP budget that checks out against $6 N_{\text{active}} D$ — none of it is exotic, all of it is the right lever for a trillion-scale reasoning model. The self-speculative decode is a real, lossless serving win. - **The MIT license is the genuinely useful part.** Weights, tokenizer, and eval harness, commercial use allowed — that is more open than most models at this scale, and it means the benchmark claims can, in principle, be checked by anyone with the hardware. - **The benchmarks are the part to hold at arm's length.** Saturated, self-reported, own-harness scores are a marketing artifact until someone independent reproduces them. I would love to be wrong; I would rather wait for the third-party numbers than quote 99.4 as if it were settled. The bet Monolith makes is that a trillion-scale open MoE, routed and staged carefully, can be a frontier reasoning model in the open. The architecture is a credible version of that bet. Whether it actually reasons at 99.4-on-HLE levels is a question its own harness cannot answer. --- *Sources: the [Monolith 1.0 model card](https://huggingface.co/basaltlabsai/monolith-1.0) and the [Basalt Labs tech report](https://basaltlabs.org/monolith) (architecture, training, deployment, benchmarks). Benchmark numbers are quoted as reported by Basalt on their own harness at maximum thinking effort. The training-compute figure is checked against $C \approx 6\,N_{\text{active}}\,D$ with the reported 49.5B active parameters and 60T tokens. The interactive diagrams illustrate the mechanisms; the routing, context, and decode visuals are illustrative, and the 65,536-token YaRN midpoint is my own clean two-stage split, not a disclosed intermediate length.*