NodeWright: Kubernetes-Native Package Manager for GPU Node Fleets

NVIDIA has introduced NodeWright, an open-source, Kubernetes-native package manager for declaratively configuring and updating host operating systems across GPU clusters. The project, previously known internally as Skyhook, is now available as open source and is part of the NVIDIA DSX OS software layer. NodeWright manages the full lifecycle of host-level changes, from installation through configuration, upgrade, and uninstallation, while respecting Kubernetes primitives like PodDisruptionBudgets, node selectors, taints, and tolerations.

How NodeWright works

NodeWright consists of three main components: an operator, custom resources, and packages. The operator is a Kubernetes controller that watches for NodeWright custom resources and manages the lifecycle of changes across nodes. Packages are container images that carry scripts, configurations, and binaries, along with verification scripts that surface failures and stop rollouts when modifications are incorrect.

When a NodeWright custom resource is applied, the operator orchestrates a sequence on each targeted node: cordon the node, wait for critical workloads to finish, drain remaining pods, apply and configure packages, interrupt if needed (e.g., restart a service or reboot), and uncordon the node. Pods labeled as non-interruptible keep the sequence in the wait stage, allowing running training jobs to finish or move before interruption.

Packages can perform host-level operations that typically require root access, such as setting sysctl and GRUB parameters, configuring crash dump collection, creating logical volumes, installing security agents, and remediating CVEs. NodeWright tracks the state and semantic version of every package on every node, distinguishing between fresh installation, upgrade, and downgrade. Packages can declare dependencies, and validation checks can be paired with apply, configuration, upgrade, uninstall, and post-interrupt work to verify expected node state and surface failures through Kubernetes.

New nodes can join the cluster with a taint, complete required package operations and validation checks, and only have the taint removed after passing, ensuring new capacity does not accept production workloads until ready.

Safe rollouts at scale

The NodeWright DeploymentPolicy resource provides progressive rollout strategies. Compartments, named groups of nodes selected by labels, each have their own disruption budget and rollout strategy. Three strategies are available: fixed (constant batch size), linear (increase batch by a fixed delta), and exponential (multiply batch by a growth factor). Each strategy includes a batch threshold, the minimum success percentage required before advancing, and an optional failure threshold that stops a compartment if too many consecutive batches fail.

This allows a fleet-wide kernel update to start with a single canary node, verify health, and accelerate automatically. If something goes wrong, the update stops rather than cascading. NodeWright reports errors in its status, marks failed Jobs, and adds a label and condition to each affected node.

Integration with NVIDIA AI Cluster Runtime

NodeWright integrates with NVIDIA AI Cluster Runtime to apply host-level parts of version-locked recipes, and works alongside NVCRE and NVSentinel for pre-workload validation and runtime fault monitoring. The public package repository provides modular components for running shell commands, managing bind mounts, and establishing kernel crash dump collectors.

NVIDIA also publishes packages based on operational knowledge from running GPU clusters at scale. Tuning packages use an intent-based model: instead of naming a profile, you declare your accelerator and use case, such as NVIDIA Blackwell GPUs for multi-node training. The package automatically assembles the right profile, covering kernel parameters, power management, and system settings. Coverage spans NVIDIA Hopper and Blackwell GPUs, with a generic baseline profile for any NVIDIA GPU and variants for different cloud environments. A dedicated package covers Google Kubernetes Engine nodes running Container-Optimized OS, where the usual tuning stack is not available.

NodeWright packages are defined as Custom Resources, so they deploy through kubectl, Helm, Argo CD, Flux, or other GitOps tooling. The project is available via Helm, with a packages repository for NVIDIA and community packages, and documentation covering architecture, CLI reference, and deployment policies.