NVIDIA NIM Optimizations Lift Nemotron 3 Ultra Throughput 2.5x on 4xB200

NVIDIA reports that its NIM 2.0.12 optimized serving stack delivers up to 2.5x higher system throughput for the Nemotron 3 Ultra model on a 4xB200 system compared with a baseline serving stack without NIM optimizations. At a target of 50 tokens per second per user, the optimized stack reaches 1,997 tokens per second of system output throughput, while the baseline reaches 718 tokens per second. The company presents this as a way to serve more concurrent users at the same interactivity level, defined here as 20 milliseconds time to first token.

The benchmark uses an agentic workload with 64K context, 400 output tokens, 76 percent key-value cache reuse, and the 50 TPS per user target. NVIDIA states that the gains come from interacting configuration bundles rather than independent switches whose percentages can be added. The optimization layers include autotuned mixture-of-experts and Mamba kernels, tensor parallelism across the four GPUs, prefix and model-state reuse, scheduler and memory tuning, and multi-token prediction speculative decoding.

What the optimized stack changes

NVIDIA describes the NIM 2.0.12 stack as combining several techniques. Autotuned kernels map the hybrid architecture to Blackwell GPUs. Tensor parallelism distributes the model across four GPUs, with expert-aware execution for mixture-of-experts layers. Prefix caching avoids recomputing repeated context, and partial-prefix matching recovers reuse when only part of a prefix matches. Mamba state-cache settings are tuned for the model architecture. Scheduler, batching, and memory tuning adjust concurrent-sequence limits, batched-token limits, block size, and GPU-memory allocation to keep more work in flight without crossing the latency target. Multi-token prediction speculative decoding adds its incremental benefit depending on acceptance rate and available memory headroom.

NVIDIA positions NIM as a deployable microservice that packages model- and GPU-aware serving choices into a validated configuration. Instead of starting from a blank runtime configuration, developers get a tested starting point with standard APIs. For production, NIM Certified adds regular inference-stack updates, CVE handling, broader hardware validation, and commercial support through NVIDIA AI Enterprise.

Benchmarking against your own traffic

NVIDIA notes that the published curves are a starting point, not a promise that every application will see the same result. The company recommends replaying representative traffic and building a Pareto curve for the latency metric that matters. The suggested workflow is to deploy the exact software versions, pin the image tag or digest, prepare a Mooncake-format JSONL trace or capture controlled NIM requests, and use NVIDIA AIPerf to replay the traffic. The example command sweeps concurrency levels from 1 to 64 against a local endpoint, then compares output throughput among points that satisfy the service-level objective.

For a four-GPU B200 system, NVIDIA suggests setting the model profile to vllm-nvidia-b200-nvfp4-tp4-pp1-throughput-90.0 and enabling speculative decoding via the NIM_SPECDEC_ENABLE environment variable. The NIM is available on NGC with the 2.0.12 tag or a specific published digest, and a hosted API is available for immediate experimentation.

NVIDIA says more performance-optimized NIM configurations are planned across a broader range of models, giving developers additional validated starting points for their own latency, throughput, and cost objectives.