vLLM vs. SGLang: KV cache designs shape engine choice

Open weight models now handle most production token volume on major gateways, and teams that want control over data and costs are increasingly running those models themselves. Red Hat's Cedric Clyburn compares the two open source engines that dominate at-scale serving, vLLM and SGLang, and explains how each one's original design for managing the KV cache still shapes which workloads it handles best.

Two answers to the KV cache problem

Every LLM request has two stages. Prefill processes the whole prompt at once and computes attention keys and values for every input token. Decode generates output one token at a time, reusing those stored keys and values. That stored state is the KV cache, and it grows with every conversation, consuming GPU memory. An inference engine is essentially a scheduler and memory manager for that cache, deciding which requests run together and where cached state lives.

vLLM came out of UC Berkeley in 2023 with PagedAttention, which splits the KV cache into small fixed-size blocks that can sit anywhere in GPU memory. Memory is allocated as a sequence grows instead of reserved up front, so the same GPU fits many more concurrent requests. The original vLLM paper found existing systems wasted 60 to 80 percent of KV cache memory. Paged KV cache management is now standard across modern serving engines, SGLang included.

SGLang, introduced through LMSYS in January 2024, stands for Structured Generation Language. Its RadixAttention stores computed KV cache in a radix tree keyed by token prefix. When a new request arrives, the engine reuses everything that matches and computes only the new part. That suits multi-turn chat, extraction workflows, and agents that send the same system prompt and tool definitions on every call. SGLang pairs this with cache-aware scheduling, running requests that share a prefix close together so the cached branch is still available.

Shared capabilities and independent benchmarks

Both projects have expanded beyond their original optimizations. They now offer continuous batching, prefix caching, chunked prefill, structured outputs, speculative decoding, quantization, and distributed execution. Both support NVIDIA, AMD, Intel, and Google accelerators, with NVIDIA and AMD treated as first-class. vLLM's support for TPU, Intel GPU, CPU, and Apple is older and documented for serving.

Both projects rank in the top 10 of GitHub's 2025 Octoverse report by contributor growth, vLLM at number 3 and SGLang at number 7. In 2025 both joined the PyTorch Foundation ecosystem, with vLLM as a Foundation-Hosted Project after passing strict production-use and governance checks, and SGLang as an Ecosystem Project with independent governance.

Modal's LLM Engineer's Almanac tested vLLM, SGLang, and TensorRT-LLM on more than a dozen models and 10 input and output length combinations on NVIDIA GPUs with out-of-the-box settings. Modal's conclusion: vLLM and SGLang perform comparably, so the choice comes down to other factors like how quickly each ships needed features. For Llama 3.1 8B on a single H100, at the highest request rate where each engine still returned the first token in under 1 second at p95, vLLM hit 17.08 requests per second versus SGLang's 16.88 for a 1024 token input and 128 token output workload. For 128 tokens in and 1024 out, SGLang reached 6.50 requests per second versus vLLM's 5.34. For 2048 tokens in and out, vLLM managed 1.23 requests per second and SGLang 1.25. Time to first token at p95 stayed under a second for both engines across all three shapes.

One detail matters when reading those numbers: Modal's test prompts were random chunks of Pride and Prejudice, so very few requests shared a prefix. That is a fair test of independent requests, but it does not show the prefix reuse that matters for chat and agents, exactly where RadixAttention and vLLM's prefix caching make a difference. The results also come from a single replica with default configs, so tuning can change the picture. Red Hat's takeaway is that the best benchmark is your own traffic, and tools like GuideLLM let teams run realistic prompts against either engine on their own hardware.

The right engine depends on the workload. vLLM's PagedAttention fits more independent requests onto a GPU, while SGLang's RadixAttention reuses work across related calls. Both have adopted each other's best ideas, and both are backed by large communities, so the choice comes down to traffic patterns, hardware, and which features ship first.