NVIDIA ModelExpress Cuts Model Startup From 8 Minutes to Under 2

NVIDIA's ModelExpress (MX) is a new framework that accelerates the model weight lifecycle by choosing the fastest available source for weights before loading. Instead of treating every replica as an independent cold start, MX checks where a compatible copy already lives and transfers it directly, prioritizing GPU-to-GPU P2P RDMA via the NVIDIA Inference Xfer Library (NIXL) over redundant downloads from object storage.

In tests with DeepSeek-V4 Pro on an 8x B200 GPU node with ConnectX-7 NICs, MX transferred weights and JIT kernel cache artifacts from a serving replica to a fresh replica in under 10 seconds, reducing total startup time to 1 minute 44 seconds from 8 minutes. The framework also coordinates fleet-wide downloads so the cluster pays external ingress cost only once, then reuses the cached checkpoint for every replica.

Bootstrap paths from storage to GPU

For the first worker, when no peer is available, MX bootstraps from storage using one of several paths. It can stream checkpoints from object storage using the Model Streamer, which fetches tensor ranges concurrently across shards and pipelines remote reads with GPU placement, avoiding intermediate copies to disk. In tensor-parallel deployments, participating ranks divide the remote reads and share results, typically over NCCL, instead of each rank downloading the full checkpoint.

When a shared disk cache tier exists, such as persistent volumes in Kubernetes, the MX Model Cache Service collapses concurrent requests into one coordinated download. An atomic claim in the Metadata Store selects a downloader, while other replicas track progress and reuse the cached copy. This means 10 replicas fetching the 806 GiB DeepSeek-V4 Pro model pull roughly 8 TiB of identical data only once, not ten times.

For local storage, MX uses GPUDirect Storage (GDS) when supported, reading checkpoint files directly into GPU memory through NIXL's multithreaded GDS backend, bypassing host memory. MX detects GDS capability automatically and falls back to other strategies when unavailable. Alternatively, MX can load local checkpoints through Model Streamer, which stages through host memory but overlaps disk I/O with GPU placement and benefits from the OS page cache.

Peer-to-peer weight transfer and kernel cache reuse

Once a serving replica holds compatible weights in GPU memory, MX treats it as a live weight source. After confirming compatibility, it transfers tensors directly from source GPU to target GPU over P2P RDMA. The control plane discovers compatible peers through Redis, Kubernetes CRDs, or k8s-service metadata backends, but never handles the weight bytes themselves. NIXL serves as the default transfer engine with pluggable backends for Infiniband, RoCE, NVLink, EFA, and others.

MX computes an mx_source_id from model and runtime settings that determine tensor layout, then considers only peers with a matching ID. Before any transfer, NIXL must register GPU memory backing each tensor, which can be slow for models with tens of thousands of tensors. Two opt-in strategies reduce registration overhead: pool registration registers each underlying cudaMalloc allocation once instead of each tensor, cutting registration count by 80 to 99 percent on typical models; VMM arena registration routes all load-time allocations into a single 16 TiB virtual-address arena and registers the whole used range as one dmabuf-backed region, collapsing registration to one call total.

MX also extends the approach to kernel compilation artifacts. During first forward passes, engines JIT-compile and autotune kernels such as torch.compile, Triton, DeepGEMM, and TileLang, which can take several minutes for models like DeepSeek-V4 Pro. MX's Artifact Transfer API packages these file-backed artifacts and transfers them between replicas over NIXL CPU-to-CPU RDMA, eliminating repeated JIT warmup without requiring a shared ReadWriteMany volume.

Runtime path selection and RL weight updates

At startup, MX probes available capabilities and skips unsupported paths. The priority order is P2P RDMA, ModelStreamer, GDS, then default loader (host-staged POSIX I/O). If a path fails before modifying model state, MX falls through automatically. If failure occurs after weights begin landing, it reinitializes the model before continuing, so partially written weights are never served.

For RL post-training, MX enables receiver-driven weight refit where rollout workers pull updated tensors directly from trainer ranks, mapping distributed trainer layouts to the inference engine layout on the fly. This avoids the repeated cost of moving updated weights from trainers to rollout workers through intermediate storage.

MX is designed to be hardware and software agnostic, with platform-specific fast paths enabled only where supported. The framework has a first-class transport interface that allows libraries such as fabric-lib and standalone Mooncake to integrate with MX.