ThunderAgent Scheduler Cuts KV Cache Thrashing for Agentic Inference

Together AI and researchers from Georgia Institute of Technology, the University of Illinois Urbana-Champaign, and Carnegie Mellon University released ThunderAgent, a scheduling layer for agentic inference that treats each multi-turn workflow as a schedulable program. The system, accepted to ICML 2026 as a Spotlight paper, reports more than 2x single-node throughput and near-linear multi-node scaling for synthetic data generation workloads.

The thrashing problem

Agentic workloads alternate between GPU-heavy reasoning phases and GPU-idle acting phases, when an agent waits for a tool like a compiler. At high concurrency, hundreds of agents compete for GPU memory, and their KV caches grow with each turn. Traditional request-level engines, such as SGLang, vLLM, and TensorRT-LLM, schedule each LLM call independently, with no awareness that it belongs to a longer workflow. When an agent pauses for a tool call, its KV cache can be evicted to make room for another request, then recomputed from scratch when the agent resumes. This cascade of evictions and recomputations, which the authors call KV cache thrashing, degrades throughput and latency.

Simply adding more GPU nodes does not fully solve the problem. Multi-node routers like SGLang Model Gateway pin each agent to a fixed node to preserve cache locality, but agentic context lengths grow unpredictably. Some nodes end up with long-context agents that exhaust memory, while others sit idle. KV cache offloading to CPU memory or disk, as in LMCache and HiCache, expands total cache capacity but only delays thrashing: when the working set exceeds all storage tiers, evictions resume.

Program-level scheduling

ThunderAgent sits between agentic clients and inference backends. It abstracts each agentic workflow as a schedulable program, tracking its execution phase, KV cache footprint, and node placement. Under memory pressure, it pauses low-priority workflows to reduce the number of programs competing for cache. With fewer concurrent programs, the remaining active workflows achieve higher KV cache hit rates and lower latency. When paused workflows are ready to resume, ThunderAgent routes them through a global waiting queue to the node with the most available capacity, balancing load across the cluster.

This replaces session-based static node-pinning with a dynamic routing strategy that balances KV cache locality against multi-node workload balance. ThunderAgent also remains compatible with KV cache offloading, treating GPU HBM, CPU RAM, and disk storage as a unified pool, though its pausing and resuming strategy works regardless of storage tiers.

Evaluation results

Together AI integrated ThunderAgent into its internal synthetic data generation pipeline, the same harness used for datasets like CoderForge, where hundreds of coding agents interact with sandboxed environments over dozens of turns. On a single 8xH100 node with HiCache offloading, at batch size 192, the default SGLang scheduler achieved 390 tokens per second with a mean latency of 65 seconds, while ThunderAgent achieved 803 tokens per second with a mean latency of 10.6 seconds. The authors report up to 2.5x higher single-node throughput and roughly 10x lower P50 latency at high concurrency.

In multi-node tests against SGLang Gateway on up to 8 H100 nodes, ThunderAgent's throughput scaled from 671 to 2,248 steps per minute as the cluster grew from 16 to 64 GPUs. The speedup over SGLang Gateway widened from 1.79x at 2 nodes to 2.39x at 8 nodes, with near-linear scaling. The authors attribute the gap to SGLang Gateway's increasing memory imbalance as the cluster grows, while ThunderAgent's global waiting queue routes resumed workflows to nodes with available capacity.

Adoption and integration

ThunderAgent is open source and designed as a drop-in layer. It interfaces with inference backends through OpenAI-compatible endpoints and works alongside existing optimizations like quantization and speculative decoding. The only client-side change is adding a program_id field to identify which program each request belongs to. The system has been adopted by open-source frameworks including SkyRL and NVIDIA Dynamo. The paper and code are available on arXiv and GitHub.