EPD Disaggregation Cuts Multimodal TTFT Up to 5x, NVIDIA Reports

NVIDIA Dynamo implements encode-prefill-decode (EPD) disaggregation to separate vision encoding from LLM prefill and decode stages for multimodal inference. The technique delivers up to 5x faster time to first token (TTFT) and 7x faster end-to-end response time for image-heavy prompts, short-to-medium outputs, and quantized mixture-of-experts models, according to benchmarks published by NVIDIA on September 9, 2026.

EPD splits the vision encoder into independently scalable workers. In aggregated serving, vision encoding, LLM prefill, and decode share one worker and one scheduling domain. That design works when media processing is a small fraction of the workload. But as requests carry more images or video, vision encoding can take hundreds of milliseconds or longer, delaying its own prefill and contending with concurrent prefill and decode work. Under mixed traffic, text-only requests can also wait behind multimodal requests even though they do not need vision encoding.

Three encoder placement topologies

Dynamo supports three topologies. Aggregated serving runs a single worker per GPU that manages all stages. Colocated encoder places one or more encoder workers alongside a PD worker on the same GPU, keeping separate request queues and batching. Disaggregated encoder runs encoder workers on a separate, lower-cost GPU tier and transfers embeddings to the PD tier through NVIDIA Inference Transfer Library (NIXL). In NVIDIA's test environment, two RTX 6000D GPUs ran encoder workers while four GB200 GPUs ran PD workers. The disaggregated setup with homogeneous GPUs always underperforms the colocated encoder setup, so NVIDIA does not recommend it.

What determines the gains

EPD helps when vision encoding accounts for a significant share of request processing time or limits throughput. Input media load matters: more images, higher resolution, or video produce more visual tokens and more encoder work. Output sequence length shifts latency toward decode, so TTFT gains hold but end-to-end gains shrink as output grows. Model size and precision also matter: vision transformer compute is largely fixed, while LLM compute decreases with fewer active parameters and lower precision. Smaller, MoE, and quantized models benefit more, while large dense models see less gain. Mixed traffic benefits from EPD because isolating encoder work lets text requests skip waiting for the encoder.

NVIDIA ran benchmarks with Qwen3.5 122B A10B NVFP4 on four GB200 GPUs, with extra RTX 6000D GPUs in the disaggregated setup. The goodput SLO was inter-token latency under 100 ms. For an image-heavy workload with ten images per request and output sequence length 1024, TTFT dropped 58% with colocated encoder and 50% with heterogeneous placement. End-to-end improvements were modest because decode time was not reduced. The heterogeneous tier served 70% more traffic at the same latency SLO.

Across experiments with 5 to 50 images and output lengths from 128 to 2,048, both EPD topologies kept TTFT nearly flat while aggregated latency rose steeply with visual token load. End-to-end gains narrowed as output length grew: heterogeneous EPD went from a 20.3% gain to 5.2%, while colocated encoder shifted from an 11.8% gain to a 2.5% regression due to GPU contention. In scenarios with high output length and low image load, disaggregating the encoder compromises performance.

In mixed text and multimodal traffic, EPD reduced mean TTFT for text requests by 42.2% and for image requests by 30.8% by eliminating head-of-line blocking. Quantizing LLM weights to NVFP4 while keeping the vision encoder in BF16 increased colocated EPD goodput from 1.78x to 2.64x over aggregated serving. NVIDIA suggests enabling parallel media decoding, using an embedding cache for repeated media, and applying multimodal KV routing to route requests sharing media to workers already holding relevant KV blocks.