NVIDIA Open Sources Kubernetes Controller for GPU Cluster Validation

NVIDIA has released the Cluster Readiness Engine (NVCRE), an open source Kubernetes controller that validates GPU clusters by running real distributed workloads across topology-aware node groups. The tool aims to catch performance killers that standard health checks miss, such as a single slow GPU, a degraded link, or suboptimal routing, which can cripple a 512-GPU training job even when all nodes report healthy.

How NVCRE Works

NVCRE uses three custom resources: Certification, Workflow, and Job. A Certification names target nodes and test categories, and for each category it creates a Workflow. The Workflow applies catalog, platform, and GPU overrides, manages iteration count, and spawns a child Job. The Job runs the workload, monitors node health, and records measurements and failures. Failure attribution is per node and per category, so a hardware fault during an NCCL operation or a missed bandwidth target is pinned to the specific node.

The built-in catalog covers three domains: five NCCL communication variants, DCGM level-4 diagnostics, and NeMo pretraining with Nemotron 5 models at 8B and 56B sizes. NVCRE detects the GPU architecture and cloud platform from target nodes to set defaults like GPUs per node and NCCL environment variables. Pass or fail thresholds are expressed in Common Expression Language and must be supplied by the user; no defaults ship. Example thresholds include bus bandwidth at or above 900 GB/s, goodput ratio at or above 0.9, and average TFLOPs per GPU at or above 800.

Fault Isolation and Scheduling

The TestScale field selects the grouping strategy: intra-node, intra-rack using the nvidia.com/gpu.clique label, full-scale, or diagnose. Diagnose mode runs adaptive fault isolation, splitting failing groups until minGroupSize is reached, flagging suspect nodes. The maxConcurrent field limits concurrent jobs during diagnosis to avoid fabric saturation.

For multi-node GPU workloads, the WorkloadRun CRD handles setup: image, framework, node count, resource requests, and cleanup. The Framework field supports exactly one of torch, mpi, or exec. WorkloadRun injects a shared-memory volume, NCCL and platform environment variables, and enables NVLink scale-up networking when available. Specifying spec.gangScheduler opts pods into a gang-aware scheduler such as KAI Scheduler to avoid deadlock on busy clusters.

Integration and Deployment

NVCRE integrates with NVIDIA AI Cluster Runtime (AICR) for validated configuration recipes. AICR captures driver, operator, kernel, and system settings as version-locked recipes to prevent drift. NVSentinel provides continuous telemetry-driven health monitoring using DCGM metrics, Xid errors, logs, and cloud events, and can trigger quarantine, drain, and remediation workflows without consuming GPU time.

Installation uses the nvcrectl setup init command. An example certification YAML selects nodes with nvidia.com/gpu.present true and runs communication and training categories, such as nccl-all-reduce and nemotron5-8b. Results group by category and list failed nodes with reasons, like a threshold violation on gpu-node-07 or a hardware failure on gpu-node-12. NVCRE is part of the NVIDIA DSX OS operating layer, and contributions to catalog entries, workload adapters, or tests are accepted via the GitHub repository.

Traditional validation often requires manual NCCL manifests, rack bisects, and ticket-driven debugging, taking days and leaving capacity idle. NVCRE automates fault isolation, reducing engineering time from days to minutes, and provides a repeatable path to prove cluster readiness before production workloads land.