NVIDIA Details Nemotron 3 Ultra NVFP4 Quantization with Mixed Precision and Four-over-Six Scaling

NVIDIA published a detailed walkthrough of how it created the Nemotron 3 Ultra NVFP4 checkpoint using NVIDIA Model Optimizer. The quantized model drops from 1,121 GB in BF16 to 352.3 GB, a 3.2x compression that NVIDIA says cuts the hardware footprint in half while matching BF16 accuracy on nearly every benchmark. The post also claims up to 5.9x higher inference throughput than GLM-5.1 754B FP4 on decode-heavy workloads, though that figure is a vendor claim rather than an independent measurement.

Mixed precision per layer

A key point in the post is that not every layer in the checkpoint is stored in NVFP4. NVIDIA assigned different precisions based on each layer's sensitivity to quantization. The MoE routed experts use NVFP4, while shared experts and Mamba projections use FP8 per-tensor. Attention linears, latent MoE layers, embedding, output classification, and MTP layers stay in BF16. The KV cache is FP8, and the Mamba SSM cache is FP16 with stochastic rounding.

This mixed-precision design lets a single checkpoint run on both Hopper and Blackwell GPUs. On Hopper, which lacks native FP4 tensor cores, the serving framework converts to W4A16. On Blackwell, it uses native W4A4. NVIDIA says W8A8 would be the obvious Hopper choice but leaves too little memory headroom for Multi-Token Prediction, so W4A16 is the better fit.

Calibration strategies compared

The post walks through several scale-selection methods for FP4 quantization. Max scaling, also called absmax, sets the scale from the block's largest value, but a single outlier can compress other values toward zero. MSE scaling searches for the scale that minimizes average reconstruction error, but NVIDIA found lower MSE did not always improve downstream accuracy. In Nemotron 3 Ultra experiments, MSE calibration cut per-tensor weight error by 27.1% over four-over-six scaling yet produced no consistent benchmark gains.

For the previous model, Nemotron 3 Super, NVIDIA combined MSE-based block scaling for weights with a per-tensor FP8 sweep and dynamic max-based activation scaling. That recipe gave the best accuracy-to-size tradeoff for Super.

Four-over-six scaling

NVFP4 represents only eight positive values: 0, 0.5, 1, 1.5, 2, 3, 4, and 6. The gap between 4 and 6 causes aggressive rounding for weights in that range, sometimes over 13% error on a single value. Four-over-six scaling lets each weight block independently choose between a maximum of M=4 or M=6, whichever minimizes reconstruction error. It applies to weights and falls back to default NVFP4 on activations.

NVIDIA used four-over-six to set the FP4 routed-expert weight scales in Nemotron 3 Ultra. It raised the global per-tensor weight scale by 1.75x, and across all 49,152 projection weights in the model's 48 MoE expert layers, it cut median reconstruction MSE by 16.4% compared to standard max calibration. In the balanced 5.03-BPE setting, four-over-six delivered 98.5% median recovery relative to BF16, ahead of max scaling at 96.8% and MSE at 98.4%.

AutoQuantize and distributed quantization

NVIDIA Model Optimizer's AutoQuantize searches per-layer format assignments under a target bit budget. The search identified 5.03 effective bits-per-element as the sweet spot where AA-LCR accuracy plateaus after a 2.4-point gain. The NVFP4_FOUR_OVER_SIX_CFG configuration will be released in Model Optimizer 0.46 in July.

For the quantization process itself, NVIDIA used distributed quantization with Megatron-LM on 16 B300 GPUs, reducing total processing time from 120 minutes to 45 minutes versus a Hugging Face Transformers path. The setup used expert and data parallelism with EP=DP=16 for calibration.

Developers can try the recipes via the Model Optimizer launcher, including the Nemotron 3 Ultra YAML recipe for automated quantization and export on a Slurm cluster.