vLLM v0.28.0 adds Kimi-K3 optimizations, DeepSeek V4 sparse MLA, and tiered KV cache offloading

vLLM v0.28.0 includes 584 commits from 270 contributors, with 76 new contributors. The release targets performance for Kimi-K3 and DeepSeek V4, expands speculative decoding, and adds disk-based KV cache offloading.

Kimi-K3 and DeepSeek V4 optimizations

For Kimi-K3, the release adds Decode Context Parallel support, fused FlashKDA decode and prefill kernels, SiTU activation support for MegaMoE, and GEMM-RS for sequence parallelism. Combined all-gathers achieve a 1.5 to 3 times kernel-level speedup, and an adaptive speculative token budget delivers about 60 percent better DSpark TTFT. Optional shared-expert sharding saves about 17 GiB of memory per GPU. Kimi-K3 also runs on ROCm with the V2 model runner.

DeepSeek V4 now has sparse MLA working end-to-end for plain decode, MTP, and DSpark speculative decoding. AMD Quark NVFP4 support is included, along with reasoning-effort prompts and mappings, sparse top-k metadata kernel optimizations, narrowed eager CUDA graph regions, and ROCm enablement on gfx11 and gfx950.

Speculative decoding gains DFlash2 with local convolution and a candidate selector, DSpark confidence-scheduled verification, and async scheduling auto-enabled for draft models. Model Runner V2 matures with E/P/D disaggregation, weight offloading, multi-layer MTP KV cache support, encoder CUDA graphs, decoder token-wise pooling, attention-free models, and a thinking token budget.

KV cache offloading and new defaults

Tiered KV cache offloading now supports disk offloading, out-of-tree secondary tier managers via a module path, partial secondary-tier load results, tiering metrics, and a canonical CPU layout for parallelism-agnostic offload. This allows serving larger models or longer contexts by spilling KV cache to disk or custom backends.

New defaults raise max_num_batched_tokens from 8192 to 16384, enable prefix caching by default for Mamba models, and raise the Blackwell CUDA graph capture default to 1024. Breaking changes include bitsandbytes support moved to an out-of-tree plugin, Transformers bumped to 5.15.0, removal of the deprecated calculate_kv_scales runtime KV scale calculation, and removal of override_attention_dtype.

Hardware and model support

NVIDIA work includes FlashInfer XQA decode support on SM12x, a CuTeDSL fused query kernel on SM100, programmatic dependent launch for DSA decode kernels, a native DSA decode path for MTP=3 on SM90, GB10 fused-MoE FP8 tuning configs, and B12X dense linear backends. AMD ROCm gets a torch 2.12 and triton 3.7 stack bump, AITER and FP8 inference on GFX120x, DeepSeek-V4 on gfx11, optimized Triton sparse-MLA decode on gfx950, and a fused Kimi-K3 KDA decode kernel. Intel XPU adds a torch linear backend with blockwise GEMM, MXFP8 linear weights for the INC DeepSeek V4 model, and an XPU wheel in the release pipeline. CPU support includes an MLA backend for DeepSeek-V2/V3, a triton-cpu wheel, GPTQ and AWQ on s390x, and BF16 MoE routed through zentorch on AMD.

New models include Muse Glimmer, Ling 3.0 Flash with BF16, MTP, and parser support plus an FP8 variant and hybrid MXFP4 routed experts, Dots3 NOTE native multimodal support, and Interns2mobius. Qwen3.8 is enabled on AMD ROCm, with fused CUDA post-conv MTP decode for Qwen3.5 GDN and GDN gates aligned with speculative tokens. The Transformers modeling backend adds MLA support, hardware-agnostic model definition, logit softcapping, and a hardened multimodal path. LoRA support expands to vision towers for Gemma4 and tower/connector LoRA for Keye and Ultravox. Vision encoders get full CUDA graphs for Kimi-K2.5 and Ernie-4.5-VL, torch.compile for the Qwen3-VL encoder, and avoided long-blocking H2D copies.

Quantization work includes online MXFP4 support, online weight scales shared across TP, precision preserved in online NVFP4 expert packing, batch-invariant NVFP4 MoE via CUTLASS, KV 4-over-6 scale search, and a DeepSeek-V4 AMD Quark NVFP4 emulation kernel. Fixes cover dynamic INT8 W8A8 MoE configs, a TritonExperts crash, MXFP4 conversion for FlashInfer CUTLASS, and fused block-scale orientation.

Large-scale serving features E/P/D disaggregation with GPU-side preprocessing, disk offloading for SimpleCPUOffloadConnector, Mooncake store group semantics and tenant ID support, and transfer mode in the NIXL compatibility hash. Parallelism improvements include EPLB balancedness calculation fixes and dense multinode DP rescope.

API additions include request priority parsed from an HTTP header, session ID plumbing, count_reasoning_tokens in the streaming parser engine, content_parts on the generate endpoint, and vLLM Recipes connected to native config-based deployment. The Rust frontend gains a standalone renderer, multimodal image inference over gRPC, explicit data-parallel rank routing, and RL lifecycle control, with protobuf schemas published to Buf.

Release artifacts include Python wheels for PyPI with CUDA 13.0, ROCm, and CPU, plus Docker images for CUDA 13.0, CUDA 12.9, Ubuntu 24.04 variants, ROCm, CPU, and XPU. Pre-built wheels are available for CUDA 12.9 and 13.0 on x86_64 and arm64, plus CPU wheels for x86_64, arm64, and macOS.