Gradium TTS on Baseten cuts time to first audio to 214 ms with streaming and GPU tricks

Baseten published Gradium TTS as a dedicated deployment in its Model Library on October 13, 2026. The text-to-speech model is built for real-time voice agents, and independent Coval benchmarks from September 2026 place its median time-to-first-audio (TTFA) at 214 ms, with a tight interquartile range. TTFA measures the time between sending a TTS request and receiving the first playable audio, and 214 ms is faster than typical human speech response times, giving a voice agent room to reason or fetch context without breaking the conversation flow.

Streaming before the sentence ends

Gradium TTS does not wait for the full text input. It uses Delayed Streams Modeling (DSM), a framework from Kyutai that treats text and audio as two parallel streams on a shared clock. Both streams advance one step every 80 ms, and the audio stream runs a fixed number of steps behind the text stream. That delay gives the model a look at upcoming words, similar to how a human speaker adjusts tone and pacing based on what they are about to say. A longer delay improves voice quality, while a shorter one lowers latency.

The model starts speaking after the first few tokens, not after full token generation. Each 80 ms audio frame is produced by two transformers with separate jobs. The temporal transformer, called the reader, moves through the streams one step at a time. It takes the newest text token and the previous audio frame, then outputs a single vector describing the next frame and signals when it is ready for the next word, which keeps pacing natural when text arrives unevenly. The depth transformer, called the refiner, turns that vector into an audio frame. Each frame is encoded as up to 32 tokens, arranged so the first token captures the most important features and later tokens add finer detail. The refiner generates them in order, making the frame progressively more accurate.

This split keeps TTFA low because the larger temporal transformer, which tracks context, runs only once per step. The smaller depth transformer runs up to 32 times per step but stays cheap since it never looks back in time.

CPU bottleneck and GPU fixes

For small models like streaming TTS, the CPU often becomes the latency bottleneck. In large language models, GPU kernels run long enough that the CPU can queue the next launch before the current one finishes. In smaller models, kernels complete faster than the CPU can queue them, so GPU utilization drops and latency rises.

Gradium addresses this with two optimizations. Kernel fusion, using torch.compile and Triton, merges operations that previously required several separate GPU kernel launches into a single kernel, so each launch does more work. CUDA Graphs records the full sequence of kernels for a forward step once, then on every subsequent step the CPU issues a single replay command and the GPU driver runs the entire recorded sequence back-to-back. In Gradium's published benchmarks, enabling CUDA Graphs cut median TTFA by more than 2.5x at small batch sizes.

Cold starts for bursty voice traffic

Voice AI traffic is bursty, with demand spiking at business hours or during outages and marketing campaigns. Provisioning for peak demand leaves GPUs idle most of the day, while provisioning for average demand degrades performance during spikes. Autoscaling helps, but a new replica cannot serve calls until model weights are loaded into GPU memory, and pulling weights from upstream sources like Hugging Face or S3 adds rate limits and cross-region latency.

The Baseten Delivery Network (BDN) reduces cold start time with three mechanisms. Push-time mirroring copies model weights into Baseten-managed storage at deploy time and records a content-hashed manifest, so upstream is touched once and every cold start reads from Baseten's own storage. Three-tier caching checks node-local NVMe disk first, then an in-cluster peer cache, then the mirrored origin, delivering weights to H100 nodes at over 2 GB/s. Coordinated downloads assign each chunk of the model to one node during a burst, which shares it with peers over the in-cluster network, so origin bandwidth for a 50-replica scale-up is 1x the model size rather than 50x. Because weights arrive before the model container starts, the transfer happens outside the billable GPU window. The result is 2 to 3x faster cold starts, letting a Gradium TTS deployment follow sudden swings in call volume without customers waiting on a cold replica.