llm-d v0.9.0 adds batch gateway, latency predictor, and XPU SGLang image

llm-d v0.9.0 adds two new components and updates the vLLM base to v0.26.0 across most hardware targets. The release includes a new batch gateway image and Helm chart, a latency predictor image, and a new Intel XPU SGLang image. The router endpoint picker and disagg sidecar both move to v0.10.0, while the KV cache library advances to v0.9.0. The inference simulator bumps to v0.10.2, and the benchmark image jumps from v0.6.8.1 to v0.8.0.

New components and hardware coverage

The batch gateway, listed as llm-d-batch-gateway v0.5.0, is a new image plus Helm chart. The latency predictor, llm-d-latency-predictor 0.9.0, is also new. Both appear in the component table without previous versions, indicating first inclusion in this release. The XPU SGLang image, llm-d-xpu-sglang v0.9.0, is also new, alongside the existing XPU vLLM image that moves to v0.9.0.

Upstream model server images see vLLM v0.26.0 for the standard OpenAI, TPU, XPU, ROCm, and CPU variants. The vLLM Omni image also uses v0.26.0, while the ROCm Omni image stays at v0.24.1. SGLang moves to v0.5.16 from v0.5.13.post1. TRT-LLM appears with tag 1.3.0rc23, with no previous tag listed.

Routing, autoscaling, and documentation changes

Several pull requests address routing behavior. One change prevents incorrect load-aware routing for AMD SGLang backends. Another adds predicted latency routing for a prefill/decode disaggregated deployment with benchmark results. The router's Grafana dashboard moves into the llm-d repository. Default Prometheus alerting rules for the endpoint picker ship in this release.

Autoscaling documentation shifts from Prometheus Adapter to KEDA. The workload variant autoscaler guide now covers KEDA support and shared-cluster setup, and the endpoint picker autoscaling guide converts to KEDA. A nightly job for the optimized baseline on SGLang is added.

Documentation updates include a deploy section for the batch gateway, an inference payload processor guide, an async processor operations guide, and a GKE A4X (GB200) prefill/decode disaggregation overlay. The GKE overlays restructure into base and a4x directories, with an A4X Max / GB300 optimized DRA recipe for vLLM added. TPU v7 disaggregated P/D benchmark results and manifests are documented for agentic serving.

Infrastructure components remain largely unchanged. Gateway API CRDs stay at v1.5.1, GAIE CRDs at v1.5.0, and Istio at 1.29.4. AgentGateway moves from v2.3.3 as kgateway to v1.1.0. Envoy Gateway v1.8.1 and Envoy AI Gateway v0.7.0 are listed as new.

New adopters include Snowflake, Novita, and DigitalOcean. The release notes also mention a license scan report and status, and a new SIG for agentic inference. The release goal tracked in issue 1945 was not fully accomplished, according to the notes.