NVIDIA CC Keeps B200 Inference Within 4.3% Latency Overhead

NVIDIA published benchmark results on September 22, 2026 showing that confidential computing on eight B200 GPUs retained 96.1% to 98.2% of baseline output-token throughput for DeepSeek-R1 inference, with mean per-token latency overhead between 1.2% and 4.3% across concurrency levels 1 through 16. The company's performance engineering team ran a controlled comparison on a single DGX B200 system with TensorRT LLM, using a 32K input and 1K output sequence length, FP8 KV cache, and tensor parallelism of 8. The CC-on and CC-off runs held model, hardware, framework version, sequence lengths, parallelism, and concurrency constant, so the confidential compute state was the only variable.

Adapting to encrypted memory paths

NVIDIA explains that in B200 confidential computing, host-to-device transfers pass through a software encrypted bounce buffer because the GPU cannot directly access protected CVM memory. This breaks the usual assumption that pinned memory provides asynchronous transfer advantages, and some copies can block the calling thread. TensorRT LLM therefore selects pageable memory for affected paths instead of unconditionally using pinned memory. For device-to-host transfers, the framework moves repeated token and sampling-data readback to an asynchronous worker, preventing protected copies from blocking the main scheduler during decode.

The kernel autotuner also needed adjustment. CUDA-event timestamps produced an unstable timing signal in the tested CC configuration, which could lead the autotuner to select a slower tactic. TensorRT LLM now uses the GPU globaltimer for tactic measurements under CC while retaining CUDA events outside CC.

Multi-GPU communication changes

NVLS multicast is not available in B200 confidential configurations. Without NVLS, NCCL_SYMMETRIC cannot provide its intended multicast benefit but may still incur memory registration and cross-rank synchronization costs before using a non-multicast collective path. NVIDIA advises frameworks targeting CC to detect NVLS availability and choose communication algorithms that minimize latency for the given message size, topology, and workload characteristics.

Measuring overhead on your own workload

NVIDIA recommends selecting a workload with long input context, extended output generation, and low concurrency to expose CC overhead. High request volume can amortize fixed encryption costs by overlapping stalls with other work, making direct effects harder to observe. Teams can quantify overhead by running the same workload with CC off and CC on, then computing output throughput retained as the ratio of CC-on to CC-off tokens per second, and latency overhead as the percentage increase in time per output token.

The company positions confidential computing as a full-stack engineering effort, not a simple toggle. It notes that security configuration and inference optimization should be treated as a single deployment problem, and recommends enabling CC, attesting the environment, and benchmarking CC-on and CC-off using the exact workload intended for production.