NVIDIA Model Optimizer: from mtq.quantize to a 4.95-bit DeepSeek-R1

mdjsonmcpcopy↗ai

2026-09-26 · 18 min · quantization · inference-optimization · open-source · distillation · speculative-decoding · explainer

This week @Ryrenz posted a short spotlight on NVIDIA's Model-Optimizer: 3,902 stars, one Python interface for quantization, pruning, distillation, sparsity and speculative decoding, and the pre-quantized checkpoints NVIDIA made with it on Hugging Face, such as DeepSeek-R1-FP4 and Llama-3.3-70B-Instruct-FP4. The pitch: when a model almost fits, make it smaller.

That misses the interesting part. ModelOpt is the tool NVIDIA uses to make its own releases, so the library and the checkpoints are two ends of one pipeline. I read both ends:

• the source at commit 23355ed, the tip of main on 26 September 2026;
• the published checkpoints' safetensors headers, fetched with HTTP range requests. No weights were downloaded and no code was run.

Every number below carries a label. Measured means I counted it from the source tree or the file headers. Reported means NVIDIA (or the post) says so. Reasoned means arithmetic on those.

What is in the box

The README lists six techniques. Here is where each one lives in the tree and how much code backs it (line counts measured):

There is also modelopt/onnx/ (32,156 lines) for ONNX PTQ and modelopt/torch/puzzletron/ (19,435 lines) for heterogeneous pruning. The training techniques mostly plug into someone else's loop (Hugging Face Trainer, Megatron-Bridge, Megatron-LM, Accelerate). Quantization and export are ModelOpt's own, 52,516 lines between them (reasoned), so that is where I spent my time. The draft heads are the subject of my EAGLE-3 piece. On pruning, the README reports that Domyn cut Colosseum-355B to 260B with Minitron pruning plus distillation (reported).

mtq.quantize is three moves

The whole post-training path, as the README uses it:

import modelopt.torch.quantization as mtq
from modelopt.torch.export import export_hf_checkpoint
 
def forward_loop(model):
    for batch in calib_loader:   # your calibration set
        model(**batch)
 
model = mtq.quantize(model, mtq.NVFP4_DEFAULT_CFG, forward_loop)
export_hf_checkpoint(model, export_dir="model-nvfp4")

quantize() in modelopt/torch/quantization/model_quant.py does three things in order.

1. Convert. convert_to_quantized_model calls replace_quant_module. That walks named_children() and swaps every module whose type is in QuantModuleRegistry for its quantized twin: nn.Linear becomes _QuantLinear, one of 95 registrations in 23 files covering Hugging Face, Megatron, Transformer Engine, vLLM and PEFT layers (measured). The twin's _setup (in nn/modules/quant_module.py) registers three TensorQuantizer children: input_quantizer, weight_quantizer, and output_quantizer, which starts disabled.

The neat part is the weight. The twin registers weight as a dynamic attribute, so inside forward any read of self.weight returns weight_quantizer(weight). The module's original F.linear call runs unchanged and simply sees quantized weights. When the swap finishes, the code prints Inserted N quantizers.

2. Configure. set_quantizer_by_cfg applies an ordered list of entries to quantizer names with fnmatch, and later entries win. The NVFP4 preset (modelopt_recipes/configs/ptq/presets/model/nvfp4.yaml) is three imports:

1. disable everything;
2. enable *weight_quantizer and *input_quantizer with the NVFP4 numerics;
3. apply default_disabled_quantizers, which switches *lm_head*, *router*, *mlp.gate.*, mtp.*, vision towers, BatchNorm and Embedding back off.

The numerics file is the format, in full:

# modelopt_recipes/configs/numerics/nvfp4.yaml
num_bits: e2m1        # the 4-bit element
block_sizes:
  -1: 16              # one scale per 16 values along the last dim
  type: dynamic
  scale_bits: e4m3    # the block scale is FP8
effective_bits: 4.5   # 4 value bits + 8/16 scale bits

Then _check_weight_quantization_took_effect raises if the config asked for weight quantization and no weight quantizer is enabled. Its docstring says why: MoE models whose module names missed the wildcards used to sail through export as a checkpoint reading "quant_algo": null, with nothing quantized.

3. Calibrate. calibrate() puts the model in eval mode and runs whatever the config's algorithm field names. Both the nvfp4 and fp8 presets say max.

Inside TensorQuantizer.forward, calibration mode calls collect() and passes the tensor through untouched; quantize mode calls _fake_quantize, which rounds and dequantizes back to the input dtype (or _real_quantize, which returns packed storage). Fake quantization is what makes QAT work: the backward is a straight-through estimator (pass_through_bwd defaults to True), so you fine-tune through the rounding with an ordinary optimizer. QAD swaps the label loss for a distillation loss against the BF16 teacher.

Where the scales come from

A quantizer needs one number per scaling group, amax: the magnitude that maps to the format's largest code. Calibration is the search for it. ModelOpt offers several rules, all in model_calib.py and calib/.

Max. MaxCalibrator.collect keeps a running torch.max of the absolute maxima. Before any data flows, max_calibrate calibrates every weight quantizer directly on its weight tensor (weight_only_quantize). That way an MoE expert the calibration tokens never reach still gets a scale. Then forward_loop runs for the activations.

Percentile. HistogramCalibrator builds a 2,048-bin histogram, and compute_amax("percentile") returns the bin edge where the CDF crosses 99.99% by default. It is per-tensor only, and no shipped recipe selects it (measured). Percentiles return in the newer nvfp4_act_headroom algorithm, which sets the NVFP4 activation global amax to max(rho × anchor, upper) from the 1st and 99.99th percentiles of the per-block activation amaxes, to leave headroom for activations calibration never saw.

SmoothQuant. This path collects per-channel activation amax. It then multiplies each input channel by $s = \max|W|^{1-\alpha} / \max|X|^{\alpha}$ and the matching weight column by $1/s$. That moves activation outliers into the weights, where per-channel INT8 absorbs them. The default is $\alpha = 1.0$, and the path is INT8 only.

AWQ lite, the search. Two passes over forward_loop. The first caches, per input channel, the mean $|x|$ ($\bar{x}_c$) and the mean block-normalized $|w|$ ($\bar{w}_c$). The second tries every $\alpha$ in 0, 0.1, …, 1.0 (11 candidates at the default alpha_step of 0.1) and builds a per-channel scale for each:

$s_c = \frac{\bar{x}_c^{\,\alpha}}{\bar{w}_c^{\,1-\alpha}}$

The scale is clamped to [1e-4, 1e4] and normalized by $\sqrt{\max s \cdot \min s}$. The weight is multiplied by $s$ and the input by $1/s$, the quantized layer runs, and the mean squared error against the unquantized output accumulates.

The $\alpha$ with the least error wins, and the scale is folded into the weights with its inverse kept as the input's pre_quant_scale. awq_clip searches a clip ratio from 0.5 to 1.0 in steps of 0.05 instead, and awq_full does both.

NVFP4's own rules. For NVFP4 the block scale itself can be searched.

• mse tries all 126 positive finite E4M3 values as a block's scale code and keeps the one with the least squared weight error. A single Triton kernel does this (kernels/quantization/gemm/nvfp4_fp8_sweep.py).
• local_hessian weights that error by the block's 16×16 input second-moment matrix $X_b X_b^{\top}$, so it minimizes the layer's output error instead.

Here is what the three rules do to the values that reach the 4-bit cast:

expand

That bump is the max rule doing exactly what its formula says. Each block's largest value is sent to 6. The scale that does the sending is itself rounded to E4M3's 3 mantissa bits, so it can be off by up to about 1/16 either way, and the maximum lands anywhere from about 5.6 to 6.4 (reasoned). Values above 6 round down to 6.

NVIDIA reports what the smarter scales buy on Qwen3.5-9B under W4A4. The average drop against BF16 across MMLU, HellaSwag, WinoGrande and GSM8K is:

All reported. The same post measures this on the 27B model it shipped:

expand

NVFP4 in the code: two scales and a nibble

I covered NVFP4 as a format in the Nemotron piece. Here is the same format as ModelOpt computes it, in qtensor/nvfp4_tensor.py. For a weight matrix $W$ and a block $b$ of 16 consecutive weights along the input dimension:

$g = \frac{\max_i |W_i|}{6 \times 448}, \qquad s_b = \mathrm{E4M3}\!\left(\frac{\max_{i \in b} |W_i|}{6\,g}\right), \qquad \hat{W}_i = \mathrm{E2M1}\!\left(\frac{W_i}{s_b\, g}\right) s_b\, g$

The symbols:

• 6 is the largest E2M1 value.
• 448 is the largest E4M3 value.
• $g$ is the FP32 per-tensor scale, computed by get_weights_scaling_factor_2.
• $s_b$ is the FP8 per-block scale, computed by get_weights_scaling_factor. That function sets a zero scale to 1.0, clamps to [2⁻⁹, 448] and casts.

The 448 in $g$'s denominator is the whole trick. The block that holds the tensor's largest value gets a scale of exactly 448, the top of the E4M3 range. Every other block gets a smaller code (reasoned from the formula). So the FP32 scale absorbs the tensor's overall magnitude, and the 8-bit block scale only has to span the ratio between blocks.

The element cast, _cast_fp4, finds each value's position among the bounds [0.25, 0.75, 1.25, 1.75, 2.5, 3.5, 5] with searchsorted. Ties at 0.75, 1.75 and 3.5 are bumped up, which amounts to round-half-to-even on the grid {0, 0.5, 1, 1.5, 2, 3, 4, 6}. A sign bit makes the code 4 bits. Then (q[..., 1::2] << 4) | q[..., 0::2] packs two codes into each byte.

Here is that arithmetic on 32 weights in two blocks, with an outlier you can grow in block A:

Grow the outlier and watch block B. Its own FP8 scale code shrinks to keep its largest value near 6, so its error barely moves. INT4 with one scale across all 32 values has no such escape: in the default state it rounds most of block B to zero, 84.5% relative error on that block against 10.4% for NVFP4 (illustrative, seed 1, outlier at 6×). Per-tensor FP8 wins on error at every setting, because every value keeps 3 mantissa bits; it just costs 8 bits a weight instead of 4.5. Switch the block scale to the MSE sweep and the codes move. Some blocks pick a smaller code, clipping their maximum to 6 in exchange for a finer grid under it; others pick a larger one. That is the mechanism behind Figure 1.

On disk, each quantized linear becomes four tensors. The shapes below are for one DeepSeek-R1 expert's gate_proj, a 2048×7168 matrix (measured):

The file confirms effective_bits: 4.5: across DeepSeek-R1-FP4's 44,727 quantized linears, the E4M3 scale bytes are exactly one eighth of the packed-weight bytes, 41,103,261,696 against 328,826,093,568 (measured).

Activations get only the scalar input_scale, which is calibrated amax over (6 × 448). The YAML says type: dynamic, and the file has nowhere to keep per-block activation scales, so the kernel must compute them at inference time (reasoned from the layout).

Export: one checkpoint, three runtimes

export_hf_checkpoint calls _export_quantized_weight (in export/unified_export_hf.py) on each quantized module. For NVFP4 it registers weight_scale_2, computes weight_scale against it, packs the weight, and registers input_scale from the input quantizer's amax. Tensor names stay the Hugging Face names, so the checkpoint is the original layout plus scale tensors, and two configs:

• hf_quant_config.json: producer, quant_algo, kv_cache_quant_algo, group_size and exclude_modules.
• A quantization_config block in config.json: quant_method: "modelopt" plus compressed-tensors-style config_groups.

That single tag is what the runtimes key on: LLM(model=path, quantization="modelopt") in vLLM, sgl.Engine(model_path=path, quantization="modelopt") in SGLang, and a plain LLM(model=path) in TensorRT-LLM. There is still a legacy export_tensorrt_llm_checkpoint for the TensorRT-LLM C++ backend.

The README is honest about how thin the support matrix's evidence is: its entries come from tests/examples/hf_ptq/test_deploy.py, whose cases "are marked release and run out-of-band — no workflow currently passes --run-release", and each is a load-and-generate smoke test. My vLLM and SGLang reads cover the other side of that tag.

What is in nvidia/DeepSeek-R1-FP4

Summing the tensor sizes in the headers of all 80 shards gives 180,364 tensors and 423,605,060,024 bytes (measured). By dtype:

hf_quant_config.json names modelopt 0.23.0 as producer, NVFP4 with group_size 16, no KV-cache quantization, and 245 exclude_modules (measured):

• self_attn* in all 61 decoder layers;
• the router mlp.gate in the 58 MoE layers;
• 122 layernorms;
• model.norm, model.embed_tokens and lm_head;
• model.layers.61*, the multi-token-prediction layer.

So every attention projection is BF16. The card's line, "only the weights and activations of the linear operators within transformers blocks are quantized," is looser than the file: attention projections are linear operators inside transformer blocks too.

For the baseline I read the headers of deepseek-ai/DeepSeek-R1's 163 shards the same way: 688,574,839,360 bytes (measured), FP8 with F32 scales per 128×128 block. Its index file's total_size claims 1,369,062,772,000, about double, which is why I sum headers. Both files store the same 684,489,845,504 parameters. The FP8 original is 688.6 GB at 8.05 bits a weight; the FP4 release is 423.6 GB at 4.95 bits a weight, 1.63x smaller (measured). The card's "approximately 1.6x" holds (reported).

The ledger explains the missing half of a 2x.

• The saving: the routed experts go from 654.07 GB to 367.82 GB, 8 bits to 4.5. They are 95.53% of the parameters and carry the whole saving (measured).
• The growth: attention was FP8 in DeepSeek's release and is BF16 in NVIDIA's, so it doubled from 11.42 GB to 22.83 GB. The MTP layer went from 15.42 GB to 26.93 GB. Together those two groups grew by 22.9 GB (measured).

Leave out the MTP layer, which only matters if you serve with multi-token prediction, and the rest averages 4.73 bits (reasoned from the measured counts). The file does not say why attention was excluded rather than kept at FP8.

nvidia/Llama-3.3-70B-Instruct-FP4 is the simpler case, with nine shards and 42,709,045,632 bytes (measured):

• Exclusions: exclude_modules is just lm_head, so attention is quantized too.
• KV cache: kv_cache_quant_algo is FP8, with a BF16 k_scale and v_scale per layer.
• Coverage: 97.02% of the 70,553,706,496 parameters are NVFP4, and the file averages 4.84 bits a weight.
• Size: the BF16 source is gated, so I computed its size as two bytes a parameter: 141.11 GB, making the FP4 file 3.30x smaller (reasoned). The card says approximately 3.3x (reported).
• A quirk: weight_scale_2 is an F32 scalar for q, k, v, gate and up, but a BF16 tensor of shape [1] in all 160 o_proj and down_proj layers (measured). Harmless, and a sign the 0.23.0 exporter was not uniform.

The claims, marked reported

Accuracy. Both cards give one accuracy table and no speed numbers (all reported):

R1's baseline is DeepSeek's FP8 release, not BF16. Both checkpoints name modelopt 0.23.0, published on PyPI on 28 January 2025, with cnn_dailymail as R1's calibration set; the local-Hessian scales above arrived about 19 months later.

Speed. The repo's examples/benchmark.md gives H200 numbers (modelopt 0.21.1, TensorRT-LLM 0.15, 2,048 tokens in and 128 out, speedup normalized per GPU). For Llama3.1-8B, all reported:

The same page reports an MMLU loss of 1.50% for FP8 and 5.66% for INT4 AWQ on the 8B model, and 0.38% and 1.07% on the 70B model.

Weight-only 4-bit helps at batch 1 and hurts once the batch grows: decode is memory-bound, so fewer weight bytes pay at small batches, and at larger ones the dequantization shows. The newest post in the repo finds the same for NVFP4 on Blackwell, on Qwen3.6-35B-A3B in vLLM on 4× GB200 (all reported):

• W4A16 (weights only): slower than BF16 in 10 of 12 shapes, because a BF16 activation sends vLLM to the Marlin dequant-to-BF16 fallback, which never reaches the FP4 tensor cores.
• W4A4 (weights and activations): faster in 9 of 12 shapes, by up to 1.30x, and 0.87–0.88x at concurrency 1.
• Size: the checkpoint goes from 67 GiB to 22 GiB.
• Accuracy after QAD: 500 iterations of QAD bring IFBench from −2.6 points after PTQ to −0.3, and MMMU-Pro from −1.2 to −0.7.

Mixed precision is the other knob. AutoQuantize scores every layer's sensitivity with one backward pass and solves an integer program for the best format assignment under an effective-bits budget:

expand

What I'd take from it

• The format is 4.5 bits; the file is whatever you leave in BF16. R1-FP4 averages 4.95 and Llama 4.84. Budget GPU memory from hf_quant_config.json and the headers, not from the format name.
• The default scale rule is max. NVIDIA's newer Qwen3.8-27B checkpoint ships local-Hessian scales, which are computed once at export and cost nothing at inference. That is the cheapest accuracy in the library.
• The metadata can be wrong in either direction. The guard in quantize() exists because a missed wildcard produced a valid, unquantized checkpoint. A changelog entry fixes an export whose ignore list could "claim a layer is unquantized that the export in fact quantized". Read exclude_modules and the headers before you trust an "FP4" label.
• For throughput, quantize activations too. Weight-only 4-bit is a memory win, and a speed win only at small batches.

For runtime tricks that go below 4 bits, see runtime dynamic compression. For the KV cache side, see TurboQuant.

Sources: NVIDIA/Model-Optimizer at 23355ed (Apache 2.0), NVIDIA's model cards, and the safetensors headers of nvidia/DeepSeek-R1-FP4, deepseek-ai/DeepSeek-R1 and nvidia/Llama-3.3-70B-Instruct-FP4. Figures are reproduced from the repository's documentation for commentary; the widgets are my illustrations of ModelOpt's arithmetic.