AMD Runs Kimi-K3 MXFP4 on MI300X and MI325X with Disaggregated Serving

AMD published details on serving the Kimi-K3 model with MXFP4 weights on Instinct MI300X and MI325X GPUs, using a disaggregated prefill and decode setup. The checkpoint holds 1453.7 GiB of weights, which nearly fills the usable HBM of an eight-GPU MI300X node, about 1504 GiB. After loading the weights, only roughly 50 GiB remains for KV cache, activations, and the communication heap.

MXFP4 Without Native Support

The MXFP4 format does not have a native matrix-multiply instruction on CDNA 3, the architecture used by both MI300X and MI325X. To keep the memory savings, AMD requantizes the experts to a format the matrix engine supports. The kernel substitution uses an int4-SiTU path, specifically SiTUv2 a8w4, to run MXFP4 efficiently on gfx942.

Kimi-K3 has 24 MLA layers and 69 KDA layers, which require a recurrent state transfer during the prefill-to-decode handoff. AMD found two correctness defects during development: an RDMA KV write-race in MoRIIO and a KDA recurrent-state recycle bug. After fixing both, distinct-needle recall under concurrency is 57 out of 57 on both MI300X and MI325X.

Deployment Shape and Memory Budget

The deployment uses two prefill nodes and two decode nodes, a 2P2D configuration, with expert parallelism sharded sixteen ways (EP16). The 896 routed experts are split across 16 ranks, giving 56 experts and 84.2 GiB per GPU. The prefill pool runs TP2 times DP8 attention, meaning two-way tensor parallelism across eight data-parallel lanes. Each prefill lane holds 53.3 GiB of attention data, leaving 40 GiB of KV pool on MI325X and 13 GiB on MI300X for decode.

MoRIIO performs a one-sided GPU-Direct RDMA write of 14,098 bytes per token to transfer KV cache and KDA state. The cache size per request is 13.8 MiB for a 1024-token prompt. Decode per-token latency is 74 ms on MI325X and 82 ms on MI300X.

Throughput and Concurrency Limits

MI325X reaches its concurrency floor at 512 concurrent sequences, delivering 1235 tokens per second. MI300X plateaus at concurrency 64, limited by the product of MAX_NUM_SEQS and eight data-parallel lanes. Increasing MAX_NUM_SEQS from 8 to 32 on MI300X moves the throughput knee and lifts concurrency-128 throughput by 28 percent.

At long context, the KV pool becomes the binding constraint, causing a 3.1 times concurrency gap between the two platforms. Both platforms recall a needle at 900K tokens, but MI325X supports a max model length of 1,000,000 tokens while MI300X is limited to 900,000. In long-context agentic traffic replay, MI300X reaches a p99 of 244K tokens and MI325X reaches 414K tokens. Replay sessions run up to 103 turns on MI300X and 144 turns on MI325X, with about 90 percent prefix reuse. At concurrency 1, replay per-token cost matches capacity model predictions within 3 to 5 percent.

Why Disaggregation and Wide EP

Prefill-decode disaggregation lets each pool be tuned separately: prefill for high-throughput all-to-all communication, decode for low-latency streaming. Wide expert parallelism with EP16 is chosen because tensor-parallel expert placement would be memory-bound and require per-layer all-reduces. The 2P2D configuration is the minimal layout that can hold the model, given 16 expert ranks across an 8-GPU node.

AMD's report shows that serving a large MXFP4 model on CDNA 3 is possible with kernel substitution and careful memory budgeting, but the KV pool size and MAX_NUM_SEQS settings are critical for scaling concurrency, especially on MI300X.