Shadow Engine Recovery in NVIDIA Dynamo Cuts LLM Restart Time to Seconds

NVIDIA has introduced shadow engine recovery in Dynamo, a preview feature that restores LLM inference capacity in seconds after a failure. In a two-worker GLM-5.2 deployment on NVIDIA B200 nodes, the shadow engine resumed serving in 7.3 seconds, compared to 283 seconds for a cold restart, which is about 39 times slower. The feature keeps a fully initialized shadow engine idle on the same GPUs as the active engine, moving most recovery work off the serving path.

How It Works

The shadow engine pre-computes its CUDA context, captured graphs, NCCL and NIXL communicators, and weight mappings before parking. It does not materialize the KV cache; that is allocated only when the shadow is promoted. The GPU Memory Service (GMS), a per-GPU sidecar, owns physical GPU memory and hands out handles to inference engines. GMS uses the CUDA Virtual Memory Management API to decouple physical allocation lifetime from engine CUDA contexts. When an engine process exits, the driver releases its CUDA context, but the GMS reference keeps weight pages resident. This allows the active and shadow engines to share existing weight tensors without creating another copy in HBM.

Each worker pod contains two engine containers, a GMS sidecar, and a shared POSIX flock lock for failover coordination. The recovery sequence has four phases: steady, failure, cutover, and restarted, with roles swapping between engines. During cutover, the shadow acquires the lock, remaps weights via GMS, materializes the KV cache, and re-registers with the router. If a live engine becomes deadlocked, the Kubernetes liveness probe kills it, triggering the same lock-release path.

Requirements and Integration

Shadow engine recovery requires Dynamic Resource Allocation on Kubernetes 1.34 or newer with the NVIDIA GPU DRA driver installed. vLLM is the primary supported backend. Integrating GMS into frameworks requires a custom torch.cuda.CUDAPluggableAllocator bound to the weight memory pool. vLLM, SGLang, and NVIDIA TensorRT-LLM already have GMS integration via the allocator flag. The current preview does not support GMS for the KV cache, but support is under active development.

For inference engineers, the key takeaway is that the shadow engine footprint is small enough to coexist with an active engine on the same B200 GPU without exhausting HBM. This means you can run a warm standby without doubling memory usage, and failover happens in seconds rather than minutes. The feature is in preview, so expect changes as NVIDIA continues development, particularly around KV cache support and broader backend compatibility.