llm-d v0.8.1 Moves Multimodal and Batch to Production, Adds RL Mode

llm-d v0.8.1, published June 26, 2026, promotes multimodal serving, the batch gateway, and flow control to production status in the router component. The release also introduces a non-Kubernetes mode for reinforcement learning workloads, adds support for the Responses API, and replaces several custom images with upstream vLLM builds.

Router and Scheduling Changes

The endpoint picker image, renamed from llm-d-inference-scheduler, is now at v0.9.0. The disaggregated sidecar, formerly llm-d-routing-sidecar, also reaches v0.9.0. Both are part of the router component that now handles multimodal requests, batch operations, and flow control as production features.

New routing capabilities include predicted latency scheduling, agentic workload routing, and a FileDiscovery plugin that lets the router run outside Kubernetes, targeting Slurm or other RL environments. The Responses API support is also new in this version. DP-aware scheduling remains in a graduating state, meaning it is not yet marked production-ready.

The workload variant autoscaler moved to v0.8.0 with a CRD migration to the llm-d.ai API group and improved observability. The gateway API inference extension stays at v1.5.0, but its Helm charts now publish from the llm-d-router OCI registry.

KV Cache and Accelerator Updates

The kv-cache library is at v0.9.0 and adds heterogeneous memory allocation support, storage events, and multi-tier offloading that moves cached keys and values from CPU to storage. A Mooncake connector is also new. The filesystem backend connector wheel migrated to the vLLM 0.23.0 offload API.

GPU and ROCm deployments now use upstream vLLM images directly. The CUDA image is vllm/vllm-openai at tag v0.23.0, and the ROCm image is vllm/vllm-openai-rocm. The custom llm-d-cuda image remains available for advanced builds, but the llm-d-rocm and llm-d-xpu images are removed. CPU deployments keep the llm-d-cpu image at v0.7.0. The tokenizer sidecar aligns with vLLM v0.23.0, up from v0.19.1.

The inference simulator is at v0.9.2 and adds multimodal support, a Mooncake bootstrap option, and configurable latency. The benchmark workload launcher is at v0.7.0. The CUDA image itself is at v0.8.0 with CUDA 13.0.2, and the HPU and AWS EFA images are both at v0.8.0.

Infrastructure and Deprecations

Istio moves to 1.29.4 from 1.29.1, and kgateway agentgateway advances to v2.3.3 from v2.2.1. The gateway API remains at v1.5.1. CI now includes nightly TPU tests.

Two components are renamed: the inference scheduler becomes the router endpoint picker, and the routing sidecar becomes the router disaggregated sidecar. The debug CUDA image and the CUDA GB200 image are removed. The XPU and ROCm custom images are also removed in favor of the upstream vLLM builds.

The release notes point to v0.8.1 as a patch, with the branch for guides updated to release-0.8 and the inference-perf version pinned. vLLM v0.23.0 is confirmed by two contributors.