2026-08-03 · 13 min · llm · mixture-of-experts · amd · rocm · explainer
Every big open MoE release from the last two years shares one unstated assumption: it was trained on NVIDIA GPUs. AMD's Instella-MoE breaks that assumption on purpose. It is a 16B-total, 2.8B-active Mixture-of-Experts model, and every stage of it — pretraining, mid-training, long-context extension, SFT, DPO, RL — ran on AMD Instinct MI300X and MI325X GPUs under ROCm, with nothing borrowed from a CUDA cluster. AMD released six checkpoints, one per stage, plus the training, inference, and RL codebases, under the tagline "fully open" — a word this piece is going to hold them to.
Two things make this worth a full read rather than a spec-sheet skim. First, the architecture has a real new idea in it: Gated MLA, a per-token gate on the attention output, which turns out to be the same move Kimi K3 landed on independently. Second, the systems story — FarSkip-Collective — is a genuinely interesting trick: it makes MoE training and serving faster by deliberately feeding the model stale, outdated activations. That sounds like a bug. It is the entire point.
Six checkpoints, all the way down
Most "open" model releases mean one thing: a final safetensors file and a model card. Instella-MoE
releases the entire pipeline — a checkpoint at every stage, not just the one you'd chat with:
Instella-MoE-16B-A3B-Pretrain → -Midtrain → -Base → -SFT → -DPO → -Think. Every one of
those six ships with its own weights, training config, and a named, token-counted data recipe. That
is a materially different claim from "we released the weights." Pick a stage below and see exactly
what shipped for it:
~100B tokens (Dolma3 Longmino) then 37.32B annealing tokens (math/code/reasoning) — 4K → 64K via YaRN + document masking
Click through the six stages: every one ships its own weights, training code, YAML config, and named data mixture — not just the final Think checkpoint. That is what separates it from an “open-weight” release, where only the last set of weights comes out. The gaps at the bottom do not move between stages — compute, cluster size, and wall-clock time are withheld everywhere, and the technical report is still “coming soon.”
AMD draws its own line here, and it is a useful one: in the blog post the fully-open bucket is OLMo-3, SmolLM3, and OLMoE — full data, full code, checkpoints along the way — while Moonlight-16B-A3B, Qwen3.5, and Gemma-4 get filed as "open-weight": you get the final weights and usually a report, not the recipe. Instella-MoE places itself in the first group, and the checkpoint list above is the receipt.
What is genuinely missing, and it matters: no training compute figure, no cluster size, no wall-clock duration anywhere in the repo or the blog. "Fully open" usually implies you could, in principle, reproduce the run — and the one number every reproduction attempt needs first is exactly the one AMD didn't publish.
The shape of the model
Strip away the training story and here is what actually runs at inference time:
| Total / active parameters | 16B / 2.8B |
| Layers | 27 decoder layers |
| Hidden dimension | 2048 |
| Attention | Gated Multi-head Latent Attention (Gated MLA) |
| MoE routing | 2 shared experts (always on) + 6 of 64 routed experts (top-6) |
| Pretraining objective | next-token + Multi-Token Prediction |
| Tokenizer / vocabulary | DeepSeek-V3 tokenizer · 128,896 tokens |
| Context | 4K pretrained → 64K via YaRN + document masking |
| Training frameworks | Primus (Megatron-LM based) · Miles (RL, SGLang + Slime) |
| Inference | SGLang v0.5.9 with FarSkip-Collective overlays |
| Hardware | AMD Instinct MI300X + MI325X, ROCm |
The MoE layer is a shared-plus-routed design: 2 experts run on every token no matter what (the model's general-purpose knowledge), and a router picks 6 more out of 64 candidates per token — the same top-k routing and shared-expert idea DeepSeek-MoE popularized, applied at a 16B/2.8B ratio. The pretraining objective adds Multi-Token Prediction on top of ordinary next-token loss, DeepSeek-V3 style — training the model to predict a short run of future tokens, not just the next one, which both improves the base model and hands you a natural draft head for speculative decoding later.
Gated MLA: a per-token filter on attention
Standard Multi-head Latent Attention compresses the KV cache into a small latent vector, then reconstructs keys and values from it — cheap to store, same attention math otherwise. Gated MLA adds one more piece: after attention produces its output, a dedicated linear projection reads the input token and produces a gate, one value per channel, and that gate multiplies the attention output before it goes through the final output projection.
Concretely: attention answers "what did this token look up." The gate answers a second, separate question — "how much of what it found is actually worth keeping" — and answers it per channel, per token, learned from data. AMD's own framing is direct: the gate lets the model "selectively attenuate low-utility attention responses for each token." Attention decides what to look at; the gate decides how much of the answer to trust.
Every channel starts at its raw MLA magnitude (muted bar). The gate is a linear projection of the token itself, one value per channel, applied by multiplication before the output projection — the accent bar is what actually reaches the rest of the model. Switch tokens and the pattern of what gets kept changes, because the gate is input-conditioned: it decides per token which attention channels were worth attending to and turns the rest down. Instella-MoE’s gate is a single linear layer; the same idea shows up as a heavier, full-rank version in Kimi K3’s Gated MLA — two labs landing on the same attention-output gate independently.
The reason this is worth pausing on: the same idea shows up independently in Kimi K3, which also augments MLA with an input-dependent output gate — Moonshot's version is a heavier, full-rank gate; AMD's is a single lightweight linear projection. Two labs, thousands of miles apart, training on different hardware stacks, converged on "put a learned gate after MLA's output" as a cheap way to buy expressivity. When two independent teams reach for the same fix, that is usually a sign the fix is addressing something real in the base mechanism, not a one-off trick.
FarSkip-Collective: paying with staleness to buy overlap
Here is the problem FarSkip-Collective solves. In expert-parallel MoE training, each MoE layer's routing decision depends on that layer's own, freshly-computed attention output. Once the router picks experts, the tokens have to physically move across GPUs to wherever their chosen experts live — an all-to-all collective. That communication cannot start until the fresh activation exists, and the expert compute that follows cannot start until the communication finishes. Compute and communication are chained, not parallel, and on a large expert-parallel cluster that chain is expensive: the GPUs sit idle every time the network is busy, and vice versa.
FarSkip-Collective's fix is to break the dependency that causes the chain. Instead of routing on the fresh activation, it deliberately routes the MoE (and attention) sub-blocks on an outdated, partial activation — a slightly stale copy of the signal that was already available earlier. Stale data has one property fresh data doesn't: it's already sitting there, so the communication that depends on it doesn't have to wait for this layer's compute to finish. It can start alongside that compute instead of after it.
In standard MoE, dispatch and combine sit between compute stages and block the engine while tokens cross the network — the top lane goes idle every time the bottom lane is busy. FarSkip-Collective feeds the MoE sub-block a deliberately outdated, partial activation instead of waiting for the fresh one, so dispatch no longer has to wait on this layer’s own compute to finish — it can start alongside it. Combine works the same way in reverse: the engine moves on to the next layer’s attention before the collective has actually landed. Stale is the point — tolerating a slightly old input is what buys the overlap. (Illustrative units; AMD reports +12.7% pretraining throughput and −39.2% TTFT from the real implementation.)
That is the whole trick, and it generalizes past this one model: the separate FarSkip-Collective paper (Dukler et al., MLSys 2026) reports 97.3% prefill communication-computation overlap and 88.9% training all-to-all overlap, validated on models from 16B up to 109B parameters — including converting Llama 4 Scout to the FarSkip architecture via self-distillation and landing within 1% of the original's accuracy. For Instella-MoE specifically, AMD reports the trade paid off exactly as advertised:
+12.7% pretraining throughput from overlapping expert-parallel communication, and up to 39.2% lower Time to First Token when serving with expert parallelism — a systems win that costs nothing in serial correctness, because the model is trained end-to-end to expect stale inputs at those points rather than having staleness bolted on after the fact at serving time.
Where it lands
AMD ran its evaluations through OLMES, Allen AI's open evaluation harness — a real third-party framework, even though AMD is the one running it. On standard benchmarks, the base checkpoint lands second among six comparably-sized models, ahead of every "fully open" peer:
Instella-MoE-Base runs at 2.8B active parameters — less than every model above it except
Moonlight, and well under OLMo-3-7B's 7B dense parameters. It also leads all six on
WinoGrande at 86.5, and posts HumanEval+ 65.7, a solid coding number for a base
checkpoint that hasn't seen SFT yet.
Long context is where the honest counterexample lives. At 64K tokens on HELMET and RULER, the dense 7B OLMo-3 actually wins:
| Model | HELMET avg | RULER avg |
|---|---|---|
| OLMo-3-7B (dense) | 43.1 | 80.2 |
| Instella-MoE-Base | 41.5 | 79.4 |
| SmolLM3-3B-Base | 37.6 | 78.6 |
AMD reports this without burying it. A sparse 2.8B-active model narrowly losing to a dense 7B on long-range retrieval is a plausible, checkable result, not a suspicious one — and it's a useful reminder that "active parameters" isn't the only variable that determines long-context strength.
After SFT, the post-training funnel adds up:
SFT → DPO → Think is a steady climb, not a single post-training jump, and the RL stage's gain is
concentrated exactly where you'd expect from an instruction-following-focused reward: IFEval moves
from 77.08 after DPO to 83.70 after RL, the single largest jump on the sheet. The RL recipe
itself is worth a note for anyone who followed Rollout Routing Replay:
Instella-MoE's IF-RL stage uses R3 alongside GRPO/DAPO-style tricks (zero-gradient filtering, active
sampling, token-level loss, no KL term) — the same fix for MoE-RL's rollout/training routing mismatch,
here as one ingredient in a larger recipe rather than the whole story. A second RL stage,
Multi-Teacher On-Policy Distillation, then anchors the model back to both an IF-specialized teacher
and the DPO checkpoint, so the instruction-following gain doesn't come at the cost of the math and
code ability DPO already had.
What's still closed
"Fully open" is doing real work as a claim here, so it deserves the same scrutiny as any benchmark number.
What's released, precisely: all six stage checkpoints on Hugging Face; the full training
codebase (Primus-based, Megatron-LM lineage) under MIT; the inference overlays and RL codebase
(Miles, built on SGLang); per-stage YAML configs; and named, token-counted data mixtures for every
stage, down to the individual sub-corpus (Nemotron-CC-Math-v1, Dolma3 Dolmino, cranecode,
instella-gsm8k-synthetic, and dozens more).
What isn't released: the model weights carry a Research RAIL license — academic and research use only, not the permissive MIT the code ships under. So the license itself is split: the code is as open as it gets, the weights are open-but-restricted. Beyond licensing, three things are just missing: training compute and cluster size (undisclosed anywhere), wall-clock training duration, and a peer-reviewed technical report — the citation AMD gives today is for a different, earlier dense 3B Instella model plus the separate FarSkip-Collective systems paper, not an Instella-MoE-specific report. The blog itself says the report is "coming soon."
There's a methodology gap worth naming too: it isn't stated whether the comparison models' scores (Qwen3.5, Gemma-4, Moonlight, SmolLM3) were re-run by AMD under the same OLMES harness, or taken from those models' own published numbers. Either is defensible, but the blog doesn't say which, so treat cross-model comparisons as directionally trustworthy rather than exactly apples-to-apples.
What training off NVIDIA proves, and what it doesn't
The part of this release that will get repeated the most is also the simplest to state: a competitive, 16B-parameter MoE, trained through every post-training stage including RL, ran entirely on AMD Instinct hardware. That is a real data point. It says the ROCm software stack — Primus for pretraining, Miles and SGLang for RL and serving, FarSkip-Collective's overlays making expert parallelism efficient on this hardware specifically — can carry a full modern LLM pipeline, not just a pretraining demo.
It does not say AMD hardware is cheaper, faster, or as mature to develop against as the CUDA ecosystem for this workload — none of the numbers that would let you compute a cost-per-FLOP or a wall-clock comparison are published. It does not say the result would look the same at 10× the scale. And it's one vendor's own benchmark of its own model on its own hardware, run through a credible third-party harness but not independently reproduced elsewhere yet. What it does rule out is the null hypothesis that this can't be done at all outside NVIDIA — six checkpoints and a working RL pipeline are hard to argue with on that specific point, even while the cost question stays open.
The take
Two real ideas, evaluated honestly, and a rare complete-pipeline release: Gated MLA is a cheap, convergent fix (the same one Kimi K3 found independently) for getting more out of MLA's compressed attention; FarSkip-Collective is the more interesting systems idea, because it's not "make the network faster" — it's "make the network's timing not matter" by feeding the model activations that are already a step behind, on purpose. Together with a genuinely complete checkpoint trail — pretrain through RL, not just a final drop — Instella-MoE beats every other "fully open" peer AMD names and trails only a larger, open-weight-only Qwen3.5-4B. The unresolved part is exactly the part AMD chose not to publish: what the whole run cost, on how many GPUs, for how long. Until the technical report lands, that number stays the reader's to estimate, not AMD's to claim.
Sources: the Instella-MoE GitHub repository (architecture, training stages, license, data preparation), the ROCm technical blog (benchmarks, FarSkip-Collective and Gated MLA framing, figures), the Hugging Face model collection (six checkpoints), and the FarSkip-Collective paper (Dukler et al., MLSys 2026 — overlap percentages, cross-scale validation, Llama 4 Scout conversion). Figures reproduced here are the blog's own Figures 1–3. Benchmark numbers are AMD's, via OLMES; the training-cost and cluster-size figures this piece flags as missing are missing because AMD has not published them, not because they were left out here. Interactive diagrams are mine; the FarSkip timeline and gate values are illustrative, not measured traces.