Together GPU Clusters add passive health checks, auto repair, and OIDC

Together AI introduced a set of changes to its GPU Clusters service on July 15, 2026, aimed at the operational realities of running training and inference at scale. The updates fall into two themes: platform health, which includes passive health checks, auto node repair, and a rebuilt Slurm-on-Kubernetes stack, and operational control, which adds a new cluster details view, external OIDC, startup scripts, and an acceptance-test opt-out.

Catching and fixing failures as they happen

The company already ran active health checks, which are synthetic tests that exercise hardware against a known-good baseline. Active checks work well at provisioning time and on idle nodes. Passive health checks extend coverage to failures that appear while real workloads are running. They work like alerts and run continuously across every node, observing live workloads, logs, and metrics to surface degradation as it happens. The coverage list includes GPUs falling off the bus, thermal throttling, Xid errors, Slurm node drains on failure, and an expanding set of hardware and software failure signals. The checks observe live workloads with near-zero overhead on running jobs.

Detection alone is just a dashboard, so Together paired passive checks with auto node repair. When the monitoring system detects a node-level issue, it generates a recommended remediation and surfaces it for an operator to review. There are four repair actions, mapped automatically based on the failure signature: reboot, which preserves local data and is the default for transient issues; reprovision, which rebuilds the node from a clean image and clears local data; failover, which moves the workload to a fresh bare-metal node and clears local data; and remove, which pulls the node out of the pool and sends it to RMA. The company describes the approach as human-in-the-loop: the system detects and recommends, the operator approves, and Together handles the graceful recovery. Fully automated node-repair options are in development for specific failure modes, but for now the priority is keeping production workloads uninterrupted.

Together also rebuilt its Slurm-on-Kubernetes stack from the ground up, based on its fork of the open-source project Slinky. The upgraded stack includes self-healing worker daemons that auto-restart in place, automatic reaping of orphaned processes, durable job accounting backed by PVCs instead of ephemeral storage, reliable process cleanup that tracks every descendant of a job at the kernel level, and accurate GPU state after reschedules by rebuilding Slurm's GPU view fresh on every node start. The new stack also exposes DCGM metrics in the cluster's Grafana dashboards. All newly provisioned Slurm clusters run on the latest stack by default, and existing managed Slurm clusters can be migrated in place at no cost via a maintenance window.

Operational control

As clusters grow across teams, access, visibility, and customization become operational problems. Together rebuilt the cluster overview page around three questions: Is it healthy? Is it being used? What's happened to it recently? The new view shows node health at a glance, live usage metrics across all GPU nodes, an event timeline of node state transitions, and the cluster's full configuration in one place. Three new tabs sit alongside the overview: Nodes, Health checks, and Repair, the latter two providing historical timelines for incident retro questions.

External OIDC support for Kubernetes RBAC lets teams configure a cluster to authenticate against an existing identity provider, including Google, Okta, Auth0, Microsoft Entra ID, or another OIDC-compatible provider. Each team member runs kubectl with their own identity, the API server validates the token against the IdP, and standard Kubernetes RBAC controls access. The admin kubeconfig becomes a break-glass tool, and joiners and leavers are handled in the IdP. External OIDC must be configured at cluster creation time.

Startup scripts let users customize Slurm worker nodes, login nodes, and the controller via shell scripts that fire at specific lifecycle events: at node boot, at job start, and at job end. Scripts are configured in the Together Cloud console and can be applied to new clusters at creation time with no rebuild required. Together validates the scripts for errors.

Finally, Together changed the default for acceptance testing, which validates GPU health, networking, and storage. For smaller or short-lived clusters, the company now skips the acceptance test battery by default and goes straight to availability. For larger clusters or long-running training jobs, the calculus flips, and Together recommends enabling acceptance testing at cluster creation.