NVIDIA Dynamo Adds Session IDs to Route Agentic Inference

PyTorch and NVIDIA have updated Dynamo, their inference serving framework, to handle agentic workloads by introducing a unified session-level identifier. Agentic workloads are characterized by large initial prefill and multiple model calls with the KV cache resident between tool executions. Dynamo now makes request-level serving program-aware by assigning a stable session ID to all LLM requests in a single reasoning chain. Child sessions can include a parent_session_id to link subagents to their parent, and Dynamo automatically maps existing coding-agent headers to its internal session fields. For example, Claude Code uses x-claude-code-session-id and x-claude-code-parent-agent-id headers, Codex uses thread-id and x-codex-parent-thread-id, and OpenCode uses x-opencode-session-id and x-opencode-parent-session-id. Dynamo's CI tracks first-class harnesses and expands support without user changes. For harnesses lacking native session headers, three plugins translate identities to x-dynamo-session-id: an OpenAI-compatible Pi provider, Hermes middleware, and an OpenClaw provider. Custom harnesses can opt-in with a single X-Dynamo-Session-ID header, optionally adding X-Dynamo-Parent-Session-ID.

Session-Aware Routing and Admission Control

Session-aware routing groups requests by program_id (the session ID) and tracks two state dimensions: REASONING/ACTING and ACTIVE/PAUSED. The scheduler is a Rust plugin that owns a KvRouter directly, avoiding an extra proxy hop. It records real prompt_tokens plus completion_tokens per response instead of estimating from byte size. Session-aware admission control applies backpressure at tool boundaries, pausing sessions when workers are over capacity. This prevents the thrashing that occurs when the aggregate working set exceeds HBM or DRAM, causing full re-prefill each turn. Request-level KvRouter places each request based on immediate cache overlap, ignoring session-wide working set. Session-aware routing aims to maximize KV reuse and reduce evictions for agentic workloads, and the approach also benefits RL post-training workloads that exhibit similar multi-turn patterns.

Trace and Replay Tools

Enabling DYN_REQUEST_TRACE=1 makes Dynamo emit lightweight request_end records after each response stream. These records include session fields, output token count, finish reason, KV cache metrics, tool-call names, and replay block data. The replay block contains input length and per-block sequence hashes for content-free replay. Two replay tools are available: dynamo.replay for offline simulation and AIPerf for live GPU benchmarking. Offline AISimulate runs the captured request graph against a simulated scheduler, router, and KV cache. Live AIPerf replays the same graph on a real Dynamo endpoint to measure latency, throughput, and cache usage.

Cache Policies Across Memory Tiers

A shared-pool KV cache indexer lets the router consider reusable KV blocks stored in external caches. The Session-Prefix Indexer and proposed KvHint interface enable cache policies across memory tiers. Dynamo supplies policy intent; inference engines enforce execution according to KvHint directives. This separation allows Dynamo to optimize for session-wide KV reuse while leaving the actual memory management to the engine. For inference engineers, the practical takeaway is that agentic workloads no longer need to be treated as a series of independent requests; the session ID provides a stable handle for routing, admission control, and cache reuse, which can reduce re-prefill overhead and improve throughput in multi-turn reasoning chains.