NVIDIA Confidential Computing on Blackwell Shows 1% to 8% Inference Overhead

NVIDIA published benchmarks on July 2, 2026 showing that enabling Confidential Computing on Blackwell GPUs costs roughly 1% to 8% in throughput and time per output token during steady-state inference. The company tested the Qwen 3.5 397B-A17B model at FP8 precision on an HGX B300 with Intel TDX, comparing a virtual machine with GPU passthrough and Confidential Computing off against the same setup with it on.

How the security layer works

Confidential Computing on Blackwell GPUs, including the RTX PRO 6000, HGX B200, and HGX B300, protects enterprise data, proprietary model weights, and the model itself during active inference. The hardware root of trust is a private signing key fused at manufacturing time that is never exposed to software, firmware, or the host system. Before a confidential workload receives any secrets, the NVIDIA Remote Attestation Service verifies a signed evidence bundle, which combines the GPU's hardware report with CPU TEE measurements from AMD SEV-SNP or Intel TDX, against a reference integrity manifest. Once the confidential VM is verified, secrets such as model decryption keys can be deployed. The attestation handshake is typically a one-time startup event, so it does not add latency to individual inference requests.

Optimizations that cut secure launch overhead

NVIDIA attributes the performance impact to two main areas: secure work submission latency and reduced host-to-device bandwidth. Secure work submission latency is often the larger factor, and smaller units of work are more affected because of added encryption and kernel launch overhead. Increasing the amount of work per GPU launch reduces that impact. Workloads that depend heavily on transferring inputs to the GPU may be limited by the encrypted transfer bandwidth available in CC mode.

Three software optimizations address these costs. FlashInfer replaces event timers in CC mode with the GPU global timer register, letting autotuners compare kernel candidates accurately. SGLang moves per-step token readback off the scheduler's critical path with an async device-to-host copy worker, restoring compute and copy overlap that CC can otherwise make synchronous. SGLang also adds piecewise CUDA graph replay for prefill and mixed batches, reducing kernel launch overhead that is amplified in CC mode.

Benchmark results across concurrency levels

The benchmarks covered input and output sequence lengths of 1024/1024 and 8192/1024, with batch sizes from 4 to 256 concurrent requests. The relative performance impact ranged from approximately -1% to -8% across all configurations. For the 1024/1024 case, throughput per GPU overhead varied from -2.0% at concurrency 4 to -7.5% at concurrency 128, while median time per output token overhead ranged from -1.6% to -8.1%. For the 8192/1024 case, throughput overhead stayed between -1.0% and -3.6%, and time per output token overhead between -0.9% and -3.7%.

The test environment used Ubuntu 25.10 as the host OS with kernel 6.17.0-20-generic, Ubuntu 24.04.4 LTS as the guest OS, 256 guest vCPUs, NVIDIA driver 595.71.05, CUDA 13.2, and SGLang v0.5.12-cu130 with specific patches for SGLang and FlashInfer. NVIDIA notes that CPU power and vCPU pinning configuration should follow the referenced document. The company says it continues to work with upstream inference framework communities to keep these optimizations in place.