NVIDIA NVLink 6 Resiliency Stack Targets AI Factory Uptime

NVIDIA published a technical overview of the NVLink 6 resiliency stack for AI factories, describing how the fabric detects, contains, and recovers from transient signal errors, link degradations, and node interruptions. The company positions this multi-layer approach as a requirement for production AI infrastructure, where a single dropped packet can spike inference latency or disrupt a training collective.

Physical and link layer defenses

At the physical layer, NVLink combines lightweight Forward Error Correction (FEC), Physical Layer Retry (PLR), and Universal Physical Layer (UPHY) recovery. The lightweight FEC lets the transmitting port append error-correcting codes that the receiver uses to reconstruct corrupted bits inline, correcting single or multi-bit errors with a near-zero latency penalty. When an error burst exceeds FEC capabilities, PLR handles retransmissions directly at the physical layer, reducing packet drops to zero without involving higher-level software. If a severe degradation triggers a Link Down event, UPHY recovery recalibrates physical parameters while packets are held in a hardware replay buffer.

At the link layer, NVLink uses credit-based flow control (CBFC), where a sender only injects a packet if it holds credits indicating the next hop has buffer space. This eliminates packet loss by design and guarantees lossless transmission without the network pausing associated with Ethernet Priority Flow Control (PFC). NVIDIA contrasts this with Ethernet-based scale-up alternatives that approximate losslessness through PFC and Explicit Congestion Notification, which introduce head-of-line blocking, PFC storms, and deadlocks. The Link Manager also autonomously rebalances access and trunk links when physical links degrade, containing faults locally without cascading retransmissions or collective stalls.

Software recovery and shadow engines

The application layer provides software-driven transaction recovery executing in roughly 1.5 seconds. When link errors occur, the NMX Controller interacts with GPU drivers to place affected links into a contain and drain state, allowing hardware to retrain degraded links without data corruption or fabric-wide back-pressure. NVLink SDN control uses NMX High Availability to migrate its functional controller to an alternate switch tray within seconds if the primary host fails. The switch tray data plane is decoupled from the management CPU running NVOS, so the data plane continues forwarding even during an unplanned CPU reset or OS failure.

For uncorrectable link degradations that reach the software stack, NVIDIA Dynamo includes Shadow Engine Recovery. This maintains a fully initialized, idle replica process alongside the active inference engine, with its own pre-established NCCL and NIXL communicators. If a hardware fault interrupts the primary process, the shadow engine resumes distributed operations like Tensor Parallelism without rebuilding communicators or reloading model weights. NVIDIA reports benchmarked deployments on B200 GPUs where shadow recovery restores inference capacity in 7.3 seconds versus 283 seconds for a cold restart. The company also notes NCCL prototype support for cuda-checkpoint enables multi-node checkpoints that capture nearly all startup work, with general availability expected by the end of the year.

NVLink Fusion extends the same resiliency stack to custom XPUs, allowing third-party silicon to inherit the NVLink scale-up fabric and fault tolerance. The Vera Rubin NVL72 platform connects 72 Rubin GPUs into a single scale-up domain with NVLink 6, acting as a single unit of compute.