AMD serves GLM-5.3-FP8 on one MI300X node with DP8+EP8

AMD published a validated recipe for serving the 744B-parameter GLM-5.3-FP8 mixture-of-experts model on a single 8-GPU MI300X node using DP8+EP8 (DEP) mode with ROCm 7.2.3 and vLLM v0.28.0. The post reports that at max concurrency 128 with 128 prompts on a 96K-in / 32K-out workload, DP8+EP8 delivered 147% higher output throughput than TP=8, while TP=8 remained faster on a short 8K/1K workload. The model has 744B total parameters, about 39B active per forward pass, and a native context limit of 1,048,576 tokens as specified in config.json.

Why DP8+EP8 instead of TP=8

The default tensor parallelism (TP=8) splits every layer across all eight GPUs, which works but becomes a bottleneck at long context and high concurrency. Slicing attention across the node turns each decode step into a tensor collective, and communication overhead grows with sequence length. In DP8+EP8, each GPU is an independent data-parallel rank with its own request stream and latent KV cache, while the model's 256 routed experts are distributed evenly at 32 experts per GPU. Tensor parallelism is switched off, so Multi-head Latent Attention (MLA) runs wholly on each rank, and expert all-to-all traffic stays on the in-node XGMI fabric. The defining flags are tensor-parallel-size 1, data-parallel-size 8, and enable-expert-parallel.

Because each rank holds its own KV cache, the context window is a per-GPU HBM budget rather than a node-wide one. AMD set max-model-len to 202,752 tokens with gpu-memory-utilization 0.9, leaving room for the 96K/32K workload. The native 1,048,576-token rope length is unchanged; the shorter serving window is due to HBM constraints with TP off, where every rank keeps all 64 query heads plus a private latent KV cache and DSA indexer cache. The post notes that a full 744B-parameter replica would not fit in a single GPU's 192 GB, so the routed experts are sharded while the rest of the model is replicated.

Model architecture and memory planning

GLM-5.3-FP8 is built from 78 identical blocks. Each block uses MLA with a latent KV rank of 512, compressing full key and value tensors into a low-dimensional latent vector stored in the KV cache. DeepSeek Sparse Attention (DSA) uses a lightweight indexer to select a top-k set of past tokens, and the indexer maintains its own per-layer cache alongside the MLA KV cache, so both structures must be counted when sizing memory. The router activates 8 of 256 routed experts plus one always-on shared expert per token. FP8 applies to weights and the latent KV cache throughout the block.

Serving stack and required patches

The stack runs an OpenAI-compatible client, vLLM v0.28.0 with the V2 model runner, AITER for FP8 MLA decode and MoE kernels tuned for gfx942, ROCm 7.2.3, and 8x MI300X with 192 GB HBM3 per GPU. Two AITER patches are required: PR#4365 prevents a GPU memory fault in the FP8 MLA decode kernel when nhead equals 64 (the DP/TP=1 case), and PR#5563 fixes cold-JIT half compilation failures on the public base image. The model is roughly 756 GB across 141 FP8 safetensors shards, and a JIT cache volume of at least 10 GB must be preserved between restarts to avoid a 30-45 minute recompile. AMD notes that the native rope length is what the model supports, not automatically what a given node can serve, and has not published a full 1M-token DEP configuration on this node.