NVIDIA Exemplar Cloud: Four Fixes That Close 8-12% Training Gaps

NVIDIA published four debugging investigations from partner clusters where identical H100, GB200 NVL72, or GB300 NVL72 systems delivered 8% to 12% less training throughput than the reference architecture on the same workload, model, and global batch size. The cause was a stack of configuration choices in the kernel, hypervisor, BIOS, and NCCL settings, each costing a few percent, compounding into a gap large enough to miss the 95% threshold required for NVIDIA Exemplar Cloud validation.

Virtualization and SMMU on Grace

A GB200 NVL72 deployment running DeepSeek-V3 MoE FP8 pre-training inside a VM produced iteration times 12% to 14% longer than bare metal. Dense models like Llama 3 70B ran within 3% of reference, but the MoE workload, which issues many small kernels per iteration, was the outlier. Nsight Systems traces showed higher CPU overhead for tiny kernel regions, and a 30-second perf record with a call graph surfaced arm_smmu_cmdq_issue_cmdlist as the top frame, consuming 24% of CPU cycles.

That function submits invalidation commands to the Arm SMMU command queue. Under virtualization, every map and unmap causing guest invalidation traps the host and serializes through a single queue, producing spinlock contention. The fix was enabling VCMDQ, the Command Queue Virtualization extension to Arm SMMUv3, in the host kernel and exposing it to the guest. This requires a kernel built with the tegra241-cmdqv driver and hypervisor support. After the change, the SMMU function fell out of the top frames, dTLB miss rates returned to bare-metal parity, and the MoE gap narrowed to within reference tolerance.

CPU Power and NUMA on x86

An H100 SXM5 cluster running Llama 3 70B pre-training was 12% slower than reference. Turbostat showed busy cores pegged at 3.0 GHz despite the SKU rated for 3.8 GHz turbo, with idle cores stuck in C1 instead of dropping to C6. Numastat showed roughly 18% of memory accesses going to the remote NUMA node.

The BIOS had C-states limited to C1, a common low-latency default that is wrong for AI training. Idle cores held in C1 drew package power, so busy cores feeding the GPU could not claim enough power budget to hit turbo. Allowing idle cores to drop to C6 freed headroom, recovering about 4% on this workload. Hypervisor housekeeping threads were pinned to the same physical cores as the data loader workers, causing sporadic 50-100 ms stalls in Python threads. A cpuset separation put hypervisor and host services on cores 0-7 and 56-63, with training processes on the remainder. The 12% gap shrank to 3%, with the residual traced to a different NCCL tuning issue.

NCCL Queue-Pair Concurrency on ConnectX-8

A GB300 NVL72 deployment with ConnectX-8 SuperNICs at 1.6 Tbps per node showed a 31% gap on Nemotron-4 15B pre-training. Single-node throughput looked healthy; the gap appeared at 512 GPUs, where the profiler exposed AllGather and ReduceScatter time. Testing with nccl-tests across iteration count, UCX/UCC behavior, NUMA mapping, NVLS, and NCCL versions narrowed the fix to raising NCCL_IB_QPS_PER_CONNECTION to 4 from the default of 1.

On the reference cluster, the default configuration ran at about 1.09 seconds per iteration. With QPS set to 4, the same workload improved to about 0.83 seconds. AllGather time dropped from about 375 ms to 262 ms, and ReduceScatter from about 389 ms to 273 ms. A comparison run at about 0.76 seconds used a different NCCL version, so part of the remaining difference came from version mismatch. NVIDIA recommends keeping the deployed NCCL version unchanged and warns that QPS is fabric- and workload-dependent; on other fabrics or message-size profiles, the same setting may add CPU overhead without improvement.

Runtime Topology Propagation

The fourth case, covered briefly, involved missing NCCL topology propagation inside workload containers. Host topology files and NCCL settings correct on the node were absent in the container or launcher environment, so the collective path did not match the fabric. The fix was bind-mounting topology files into the container, which closed the gap to within reference tolerance.

NVIDIA notes these patterns are not the only causes of training performance gaps, and checking them does not replace validation with real applications. The right starting point depends on the workload, platform, and first profiler signal. Diagnostic signals came from Nsight Systems, Linux perf, and NCCL tests, and the tuning changes were verified by re-running the same benchmarks.