NVIDIA Dynamo v1.5.0 Adds Pluggable Router Policies, Sidecar, and S3 Trace Sink

NVIDIA Dynamo v1.5.0 is the 18th feature release, merging 658 pull requests from 123 contributors. The release focuses on making the router more flexible, adding a Sidecar for stock inference engines, and improving observability with native S3 trace storage.

Router and Scheduling Enhancements

Custom worker selection policies are now pluggable in Dynamo Router, using the same host selection path as built-in policies. Native scorer and picker policies run alongside the built-in selector for discovery and dispatch. Composable worker filters execute after eligibility checks and before scoring, allowing fine-grained control over which workers are considered.

Session affinity now supports hard and soft modes in Router selection, enabling sticky sessions when needed. The router also consumes Mooncake shared-cache events for SGLang HiCache without HTTP queries, reducing latency for cache-aware routing. KV events from vLLM are routed to a disk tier with locality gating, improving cache efficiency.

Short requests may bypass disaggregation and go directly to the decode stage, reducing overhead for simple queries. The frontend streams guided tool calls incrementally by default and rejects over-context requests before the first token, preventing wasted compute.

Frontend and Model Support

Qwen3 joins the unified v2 tool-call parser, and thinking controls resolve once with single precedence, applying to every chat-template key. Deployment-level defaults and an enable_thinking gate are added for Gemma 4. Bad requests now return correct HTTP status codes: 400 for validation, 503 for backend overload, and early rejection for over-context requests.

The frontend can load pluggable tokenizer backends, including Baseten's, with Hugging Face fallback. Encode-prefill-decode deployments decode images once at the frontend using libjpeg-turbo and FFmpeg, and image pixels transfer to the encode worker via a single NIXL transfer. Qwen video inputs use video-aware KV hashing, and vLLM workers accept audio and video inputs. vLLM-Omni adds realtime audio streaming and dynamic LoRA for aggregated workers.

Infrastructure and Deployment

The Kubernetes CRD storage version switched to nvidia.com/v1beta1, dropping v1alpha1 examples. The Go Endpoint Picker is removed; the Rust Endpoint Picker becomes default and ships inside the Frontend image. Dynamo Snapshot moved to a standalone operator and chart, version 0.1.0, with PodSnapshot CRDs.

GPU Memory Service V1 is introduced for checkpoint lifecycle and engine integration with SGLang and vLLM. The operator now supports runtimeVersionOverride, feature gates, explicit multinode roles, and worker canary health checks. Request migration and cancellation traces are written to a native S3 sink. Model readiness metric gates batch dispatch, and worker liveness probes now tolerate three failures.

AIConfigurator is renamed AISimulate, released as a separate package version 0.12.0 with a single CLI. AISimulate offline replay produces a single ReplayReport, with per-request capture optional.

New Recipes and Experimental Features

New recipes are added for Qwen3.8-2.4T-A95B FP8, Qwen3.5-122B FP8 on H200, Nemotron-3.5-Lightning, Nemotron-3-Ultra refresh, and GLM-5.3-Flash on vLLM. DeepSeek-V4 recipe is parameterized by checkpoint; DeepSeek-V3.2 FP4 and Llama-3-70B recipes are removed.

TLS now encrypts TCP request/response and NATS transport, with mutual TLS optional for both. The operator injects certificate paths via InfrastructureConfiguration resource. Administrative Backend SDK routes are gated on worker lifecycle, and unknown tool_call_parser names are rejected at registration.

Experimental Dynamo Sidecar runs alongside stock inference engines using native gRPC API. Sidecar images include wheel-installed launchers for SGLang, TensorRT-LLM, and vLLM in a CPU-only multi-arch image. The dynamo.triton backend enables non-LLM models to run in Dynamo pipeline via python -m dynamo.triton, with no prebuilt Triton image; build from the provided Dockerfile.

Experimental SGLang RL rollout adds finish reasons, token ids, logprobs in nvext, and native pause controls. KVBM is deprecated, with removal planned for v1.7.0. Engine versions updated: SGLang v0.5.18, TensorRT-LLM v1.3.0rc25, vLLM v0.28.0. A batch gateway example enables offline OpenAI batch requests through inference gateway, gated on Dynamo readiness. Worker liveness probes now require three consecutive failures before marking unhealthy. Request-trace records support native S3 sink for storage. The release includes 23 first-time contributors, with work on tokenization alignment and OTLP exporter.