vLLM v0.24.0 Adds MiniMax-M3, DeepSeek-V4 Optimizations, and Rust Frontend Features

vLLM v0.24.0, published on 2026-06-29, includes 571 commits from 256 contributors, with 77 new contributors. The release adds support for MiniMax-M3 and DiffusionGemma, integrates DeepEP v2 for expert parallelism, and matures the Rust frontend with new endpoints and authentication. A notable change is that vLLM no longer sets CUDA_VISIBLE_DEVICES internally; a new device_ids argument replaces it, and a deprecation window for CUDA_VISIBLE_DEVICES has begun on ROCm.

Model and Kernel Updates

MiniMax-M3 support includes BF16/FP8 indexer via MSA, MXFP4 support, FP8 sparse GQA, and extensive AMD/ROCm tuning, including mxfp8 MoE/linear on gfx950, fp8_per_channel for bf16 weights on MI300X, and an FP8 KV-cache fix. A MiniMax-M2 performance regression was also fixed.

DeepSeek-V4 received another optimization pass: a FlashInfer sparse index cache improves time-to-first-token by 2 to 4 percent, prefill chunk-planning optimization boosts end-to-end throughput by 4 percent, and a cluster-cooperative topK kernel targets low latency. Contiguous per-block KV allocations, TEP=16 for the block-FP8 shared expert, and native DSA indexer decode for next_n greater than 2 on SM100 are included. DeepSeek-V4 is now enabled on SM120 alongside GLM-5.1, with XPU and ROCm attention/MoE paths added.

Model Runner V2 (MRv2) now supports quantized models by default, enables GraniteMoE by default, and gained migration of Qwen and DeepSeek-V2 MoE models, DFlash speculative decoding, and more accurate FP32 Gumbel sampling. The new streaming parser engine unifies tool-call and reasoning parsing across models, with parsers for Qwen3, MiniMax-M2, GLM-4.7/5.1/5.2, and Nemotron V3. DiffusionGemma was added, including a CPU path and structured-output guardrails for diffusion decoders.

Other model support includes Hierarchical Reasoning Model (HrmTextForCausalLM), OpenMOSS, and Gemma 4 with unified FlashAttention across all layers. Qwen3-VL video loader, Qwen2-VL/Qwen2.5-VL processor-mapped video loader, and Qwen3-VL multi-video processing optimization were added. ViT full CUDA graph support was extended to GLM-4.1V, DeepSeek-OCR, Kimi-VL, mllama4, and Lfm2VL encoder.

Engine and Distributed Changes

Speculative decoding gained dynamic SD, DFlash with FlashInfer, mixed KV page sizes, and Qwen3Next targets. EAGLE3 support for Qwen3 was added, and TP communication for large-vocab drafts was reduced. KV cache and scheduler improvements include a KV-cache watermark to reduce preemptions, two-phase allocation for cross-group prefix-cache hits, and prefix-cache retention for Mamba and linear attention.

DeepEP v2 integration for expert parallelism includes token-bound and topk-index fixes. NIXL EP adds DBO, top-k index dtype query, NVFP4 post-receive quantization skip, and elastic-EP communicator. KV connectors for disaggregated serving include KV push from prefill to decode via NIXL, per-region KV transfer classification, and Mooncake pipeline-parallel PD support. KV offloading adds multi-tier async batched lookup, packed HMA KV-cache layout, and parallel-agnostic fs-tier cache.

Hardware and Quantization

On NVIDIA, SM90 CUTLASS FP8 mm odd-M support via swap_ab shows a 180 to 290 percent kernel speedup, and tuned fused_moe FP8 for Qwen3-Next-80B on H100 improves by 25 percent. Native DSA indexer decode on SM100 and PDL support for DeepGEMM are included. The migration of kernels to the libtorch stable ABI continued, with MoE, Marlin, Machete, and the final _C library migration completed.

AMD ROCm updates include Torch 2.11, fused AR plus RMSNorm plus per-group FP8 quant, and a fused softplus-sqrt-topk MoE router under AITER. Intel XPU gained sequence-parallel support, torch-xpu 2.12, and W4A16 int4 group_size=32 MoE. CPU improvements include 2.5 times faster ASR preprocessing via multi-threading, CPU W4A16 INT4 MoE, and cgroup memory-limit-aware KV cache sizing.

Quantization updates include online FP8 per-token-per-channel quantization, modelopt_mixed support extended to Ampere and Turing, and FlashInfer cutedsl NVFP4 GEMM backend. MXFP4 W4A4 MoE CUTLASS E8M0 scale fix, SwiGLU clamp for NVFP4 MoE on non-Blackwell, and FP8 MoE re-enabled on NVIDIA Thor are noted. GGUF quantization migrated to a plugin, and compressed-tensors WNA16 MoE actorder fix was applied.

API and Frontend

Tool calling gained strict mode in Chat Completions and Responses API. The Rust frontend added API-key authentication, CORS, /tokenize and /detokenize endpoints, /pause, /resume, /is_paused, /abort_requests, /get_world_size, and thinking_token_budget. Real /v1/embeddings support for messages and multimodal token counts in usage.prompt_tokens_details were added. Anthropic Messages API now reports cache usage and handles mid-conversation system messages. The device selection change means users must pass device_ids explicitly instead of relying on CUDA_VISIBLE_DEVICES being set internally.