Together AI breaks down what 99.9% uptime actually means for inference

Together AI published a post on July 16, 2026 that maps reliability tiers to specific failure domains and the architecture needed to survive each one. The company runs inference for teams like Cursor, Decagon, Cartesia, and Yutori, and says it has been paged for most of the failure modes described.

The post argues that reliability numbers are easy to publish but hard to interpret. GPU inference fails differently from conventional services: hardware has failure modes that CPU infrastructure does not, and systems are tuned hard for performance. Hitting 1M tokens per minute per GPU at 200 TPS, or sub-50ms time to first token on voice models with custom kernels, leaves limited slack. Adding reliability becomes exponentially harder with each nine.

What each tier requires

99% means surviving node-level failures: GPU hardware faults, driver crashes, thermal events. That typically requires automated health checking, node draining, and fast replica replacement within a single data center. The interesting engineering problem is observability. Passive health checks (hardware telemetry, metrics) give visibility without capacity overhead but miss failures that only show up under real GPU load. Active health checks catch those but require capacity to run. Together AI says it integrates checks with the scheduler so they run in the gaps between workloads, and makes the checks as quick as possible. The ceiling at this tier is the building itself: a thermal issue, a substation event, or an edge router failure can take a single-DC deployment down regardless of in-DC redundancy.

99.9% means surviving a full data center failure. That requires model weights deployed across two facilities, enough capacity on each side to absorb the full load, and live traffic routing to both, not a cold standby. The architectural decision that defines whether a provider actually delivers this tier is whether they run live traffic to both facilities continuously. Together AI chose continuous. The post also stresses infrastructure ownership: a provider renting capacity from a hyperscaler or neocloud does not own its failure domains. When something breaks at the power or cooling layer, they file a ticket with whoever does. Together AI says one ticket covers hardware, network, storage, and software because it has chip-to-token visibility across its global footprint.

99.99% means surviving a regional outage. That calls for multi-region deployment with availability zone redundancy and reserved failover capacity. The key word is reserved: not "we can route traffic there" but "we have room sitting idle there right now." GPU failure rates are meaningfully higher than CPU failure rates, and every failure mode has to be accounted for.

What to ask before committing

The post includes a list of questions for any provider, including Together AI itself. On infrastructure ownership: where are models hosted, is it single or multi-region, do you own your infrastructure or rent, who is responsible for power, cooling, and transit, and how fast can you move a deployment after an outage. On full-stack expertise: do you have chip-to-token visibility, does your team have direct hardware access, and how deep does GPU hardware expertise go. On capacity and failover: is failover tested with live traffic continuously or periodic drills, and what is the realistic recovery time objective. On SLA measurement: which failure domain does each tier cover, is the SLA measured at the load balancer or at successful inference completion, and do client-side retries count against uptime.

How Together AI measures uptime

Together AI defines delivered uptime as the percentage of time the inference endpoint successfully serves requests, measured at inference completion, not at the load balancer. It reports 99.9% consistently delivered. Its guaranteed SLA for a single data center deployment is 99%, and for a multi-data center deployment it is 99.9%. Failover time is described as seconds. Request volume is over 2B tokens per minute. A request that reaches the load balancer but fails at the GPU counts as downtime. The post also notes that availability and performance are different contracts: in Provisioned Throughput, customers pay for a GPU allocation to deliver specific tokens per second, not just for the endpoint to respond. A service that is up but delivering 30% of contracted throughput is not meeting the deal.