Run Isolated Tenant Kubernetes Clusters on Shared GPU Infrastructure

NVIDIA published a tutorial on August 3, 2026 showing how to run isolated tenant Kubernetes clusters on shared GPU infrastructure using two open source tools: KAI Scheduler and vCluster. The pattern keeps team autonomy without splitting hardware into separate clusters. A single control plane cluster hosts a GPU pool, while each team gets its own isolated Kubernetes control plane with separate API servers, CRDs, and RBAC. The example runs three teams on one NVIDIA L40S GPU, with each team seeing only its own pods.

The two tools

KAI Scheduler is a topology-aware Kubernetes scheduler built for GPU resource allocation in AI workloads. It handles pods that specify its scheduler name, while everything else goes through the default kube-scheduler. It supports hierarchical queue management with guaranteed quotas and over-quota sharing. vCluster provisions fully isolated tenant clusters, each with its own API server, custom resource definitions, and role-based access control, while sharing the underlying nodes and hardware. The virtualized control plane is invisible to tenants, with no shared control plane nodes or in-cluster agent pods.

The tutorial uses the vCluster shared-nodes model, which fits trusted internal teams. For untrusted tenants needing node, network, and storage separation, the same pattern extends to vCluster private nodes. The demo environment runs on an NVIDIA Brev GPU instance on Nebius with one L40S, 40 vCPUs, 160 GiB RAM, and 256 GiB disk. It uses Ubuntu 24.04.4 LTS, MicroK8s v1.36.2 with the NVIDIA GPU Operator preinstalled, KAI Scheduler v0.16.4, and vCluster CLI 0.35.1.

Setting up queues and tenants

After installing local tooling and enabling MicroK8s addons for DNS and hostpath storage, the tutorial adds the NVIDIA Helm repo and confirms the GPU Operator is running with Container Device Interface enabled. It then installs KAI Scheduler with GPU sharing enabled. The next step defines a queue hierarchy: one parent queue named ml-org with a total budget of one GPU, and three child queues for the NLP, Vision, and Recommender teams. Each child queue is guaranteed 0.33 of the GPU and allowed to increase to the full GPU when others are idle. The quota field sets the guaranteed minimum, limit sets the maximum allowed, and overQuotaWeight controls how surplus is split.

KAI Scheduler creates default queues automatically on first install, which serve as fallback for pods that do not specify a queue. The tutorial then creates one vCluster per team. A critical configuration setting is setOwner: false, which disables vCluster owner rewriting. This allows KAI Scheduler's pod-grouper to walk ownership chains from jobs to pods and group workloads correctly. Each team gets its own isolated control plane, RBAC, namespaces, and CRDs, and can have cluster-admin access. Underneath, all tenant clusters share the same nodes and GPUs.

The tutorial includes steps to verify that each team only sees its own workloads. It also notes that GPU time-slicing occurs at kernel boundaries between CUDA contexts, while NVIDIA Multi-Instance GPU offers hardware-level partitioning as an alternative. The pattern scales to larger clusters with hundreds of GPU nodes and dozens of teams by scaling the node pool, queue hierarchy, and number of tenant clusters. The process from KAI Scheduler installation onward works on any Kubernetes that has the NVIDIA GPU Operator running with CDI enabled.