llm-d v0.8.0: Upstream vLLM images, multimodal and batch routing graduate

llm-d v0.8.0, published June 24, 2026, replaces its custom-built accelerator images with upstream vLLM builds for CUDA, ROCm, and XPU platforms. GPU deployments now pull vllm/vllm-openai at tag v0.23.0, AMD ROCm uses vllm/vllm-openai-rocm, and Intel XPU also moves to vllm/vllm-openai v0.23.0. The llm-d-cuda image remains available for advanced builds, while the llm-d-rocm and llm-d-xpu images are removed. The CPU image ghcr.io/llm-d/llm-d-cpu bumps to v0.7.0. The change, per pull request 1791, means operators no longer depend on llm-d-maintained images for standard accelerators.

Production features and new routing

The release graduates multimodal serving, the batch gateway, and flow control to production status in the llm-d-router. New capabilities include Responses API support, predicted latency scheduling, agentic workload routing, and a non-Kubernetes mode via the FileDiscovery plugin for RL and Slurm environments. DP-aware scheduling is listed as graduating. The router components were renamed: llm-d-inference-scheduler is now llm-d-router-endpoint-picker, and llm-d-routing-sidecar is now llm-d-router-disagg-sidecar, both at v0.9.0.

The llm-d-kv-cache library, at v0.9.0, adds multi-tier KV offloading from CPU to storage, Heterogeneous Memory Allocation (HMA) support, storage events, and a Mooncake connector. The filesystem backend connector wheel migrates to the vLLM 0.23.0 offload API. The tokenizer sidecar aligns with vLLM v0.23.0, up from v0.19.1, and the inference simulator adds multimodal support, Mooncake bootstrap, and configurable latency.

Infrastructure and CI updates

Istio patches to 1.29.4, kgateway agentgateway moves to v2.3.3, and Gateway API stays at v1.5.1. The workload variant autoscaler Helm chart and image reach v0.8.0 with a CRD migration to the llm-d.ai API group and improved observability. The benchmark image is at v0.7.0. CI adds TPU nightly tests and a deployment health-check smoke test. The release also adds Rebellions as a supported accelerator vendor and includes a proposal for an llm-d-inference-payload-processor component.

Documentation was restructured: well-lit paths are renamed to guides, guides become resources, and new pages cover the EPP flow control reference, predicted latency architecture, KV indexer, autoscaling design choices, and an agentgateway guide. The gateway-api-inference-extension Helm charts now publish from the llm-d-router OCI registry.