2026-07-04 · 7 min · llm · quantization · training · mixture-of-experts · explainer
Almost every "4-bit model" you have heard of is 4-bit after the fact: the network is trained in BF16, and then a post-training quantizer compresses the finished weights so they are cheaper to serve. Training itself stays in 16-bit, because the gradients are delicate and 4 bits is a very small number. NVIDIA's Nemotron does something more aggressive: it runs the actual training GEMMs — the big matrix multiplies in the forward and backward pass — natively in NVFP4, a 4-bit floating-point format, while the model is still learning. The headline is a training-loss gap under 0.4% against a BF16 reference. The interesting part is everything that had to be true to get there.
What NVFP4 actually is
Four bits cannot, on their own, represent the range of values a weight tensor spans. NVFP4's answer is to split the job: store each value in a tiny 4-bit element, and recover dynamic range from a shared scale that a whole block of elements multiplies through. The element is E2M1 — 1 sign bit, 2 exponent bits, 1 mantissa bit. The scale is where NVFP4 differs from the MXFP4 format you may have seen: it shares one FP8 (E4M3) scale across every 16 elements, plus a single FP32 scale for the whole tensor. Flip between the formats:
Every format above stores each value in the same 4-bit E2M1 cell (1 sign, 2 exponent, 1 mantissa) — the difference is the shared scale that restores dynamic range. NVFP4 shares a fine FP8 (E4M3) scale across every 16 elements — plus one FP32 per-tensor scale — for an effective 4.5 bits/element and an E6M4-like range from 4-bit storage.
That two-level scaling is the whole trick. A 4-bit element with an 8-bit block scale over 16 values behaves, in effective dynamic range, more like a ~10-bit ("E6M4"-ish) number than a raw 4-bit one — while costing 4.5 bits per element to store (4 for the element, 8/16 = 0.5 for the scale). MXFP4 uses a coarser power-of-2 (E8M0) scale over a wider 32-element block: cheaper at 4.25 bits, but blunter, because one outlier drags a bigger block and a power-of-2 scale can only snap to coarse steps. NVFP4's finer block and FP8 scale are what make it stable enough to train in, not just serve in.
Why 4-bit training normally falls apart — and the three fixes
Post-training quantization only has to preserve the forward pass of a frozen network. Training in 4-bit is harder on two counts: the gradients flow through the same low-precision GEMMs, and any systematic error compounds over trillions of tokens. Nemotron leans on three stabilizers, each aimed at a specific failure:
- 2D block quantization on weights. Instead of quantizing weights in 1D strips, quantize them in 2D blocks, so a block's shared scale better fits the local structure of the weight matrix and fewer values get clipped.
- Random Hadamard transform on the wgrad inputs. The weight-gradient (wgrad) matmul is the one most poisoned by outliers. Multiplying its inputs by a random Hadamard matrix spreads those outliers across the block before quantization (and is undone analytically), so no single large value blows out the block scale.
- Stochastic rounding on gradients. Deterministic rounding biases small gradients toward zero — over trillions of steps that lost signal is fatal. Rounding gradients stochastically is unbiased in expectation, so the gradient direction survives quantization even when individual values don't.
Where each of these lives is easier to see than to say. Here is one path through the stack, colored by precision, with the stabilizers attached to the FP4 GEMMs — flip to the backward pass:
Only the expert weight-GEMMs — the bulk of the compute — actually run in NVFP4. Six of the seven stages shown stay higher precision: the format is most fragile at the network's edges (embeddings, the last ~16 layers, the MTP head) and on the small, sensitive projections. Flip to the backward pass to see the three stabilizers that keep the FP4 gradients from diverging.
The honest part: this is not end-to-end FP4
The precision map makes the biggest caveat visual: most of the network is not FP4. Native FP4 training means the heavy expert/MLP weight-GEMMs run in NVFP4 — that is the bulk of the FLOPs — but a meaningful fraction of the model is deliberately kept at higher precision, because FP4 is most fragile exactly there:
And it was not a smooth ride. The run diverged twice — around ~8T and ~16T tokens — and each time the team had to roll back to an earlier checkpoint and restart the segment (with an FP32-rounding fix and a re-annealed learning rate) to recover. Four-bit training is not a free lunch you turn on and forget:
The model underneath
The precision story rides on a specific architecture: a 550B-total / 55B-active hybrid that interleaves Mamba-2 state-space blocks, periodic attention, and LatentMoE expert layers, with a multi-token-prediction head on top. The Mamba-2 blocks carry most of the sequence mixing cheaply; attention appears sparingly for exact long-range recall; the MoE layers are where the parameters (and the FP4 GEMMs) live. The repeating layer pattern:
Storing that many parameters in 4.5-bit elements is a real memory win, but — honestly — not the clean 4× the "4-bit" label implies, once you count the scale overhead. Slide the parameter count:
Storing weights as 4-bit elements is not a clean 4× win: NVFP4 also carries an FP8 block scale shared over 16 elements — 0.5 extra bits each — for a real cost of 4.5 bits/param and a 3.56× shrink over BF16. And this is weight storage only: mixed-precision training keeps embeddings, projections, the final layers and the MTP head at higher precision, plus optimizer state and activations — so live training memory is larger than either bar.
For the effective storage cost per element, the formats line up cleanly — and NVFP4's 4.5 bits sits between MXFP4's leaner-but-blunter 4.25 and BF16's 16:
One more thing not to conflate
There is a second quantization result in this work that is easy to mix up with the training story: inference PTQ — post-training-quantizing the finished model down for cheaper serving. Those serving numbers are a separate experiment about deployment, measured on the trained checkpoint; they are not evidence about training precision. The claim on the table here is narrower and more interesting: that you can run the training GEMMs in 4-bit and land within a fraction of a percent of a BF16 loss curve. Keep the two apart.
The take
The quiet lesson is that "native 4-bit training" is an engineering result about where you dare to put 4 bits, not a claim that the whole network is 4-bit. NVFP4's two-level scale (4-bit element + FP8 block scale + FP32 tensor scale) buys back enough dynamic range to make the compute-heavy expert GEMMs trainable in 4 bits; the three stabilizers — 2D block quantization, Hadamard-smeared wgrad inputs, and stochastic gradient rounding — keep the gradients honest; and mixed precision quietly protects the fragile edges (embeddings, final layers, projections, MTP). The payoff is a real reduction in training compute and memory bandwidth on hardware built for FP4, at a training-loss gap under 0.4%. The caveats are equally real: it is not end-to-end FP4, a loss gap is not benchmark parity, the run diverged twice, and the inference-PTQ numbers are a different story. As a demonstration that frontier-scale training can leave the 16-bit comfort zone, though, it is a genuinely aggressive, well-instrumented bet — and it stuck.
Built on NVIDIA's Nemotron NVFP4 training report. NVFP4 is a 4-bit E2M1 element with a per-16-element FP8 (E4M3) block scale and an FP32 per-tensor scale; figures are reproduced from the paper for commentary, and the interactive diagrams are illustrations of the mechanism. Related: why quantization is hard, mixture-of-experts from scratch, multi-token prediction, and large-scale training.