SGLang v0.5.16 adds DSpark speculative decoding, Inkling support, cuts KV memory

SGLang v0.5.16 ships with a new speculative decoding algorithm called DSpark, support for the 975B-parameter Inkling multimodal MoE, and a cache layer split for GLM-5.2 that cuts per-rank KV memory by roughly 74% under prefill context parallelism. The release includes 574 pull requests from 169 contributors.

Speculative decoding and memory changes

DSpark drafts semi-autoregressively in blocks and sizes each verify window from the draft's own confidence instead of a fixed draft length. On DeepSeek-V4-Pro with tensor parallelism 8 on B300 hardware and batch size 1, the project reports 383.7 tokens per second at an accept length around 5. Operators enable it with the speculative algorithm flag set to DSPARK and a compact ragged verify mode, then tune the block with a dedicated block size option.

For ReplaySSM Ring Spec-Verify, the per-draft SSM snapshot is gone. Speculative scratch memory drops from 11.5 GB to 1.8 GB per GPU, a 6.4x reduction, on Qwen3.5-35B-A3B at tensor parallelism 1, with accuracy and throughput parity. The feature is opt-in and works only with a linear draft chain.

The GLM-5.2 DSA cache layer split shards KV and indexer cache layers across context parallelism ranks, so each rank owns a disjoint layer range instead of all layers. At 8192 tokens on GLM-5.2-FP8 with 78 layers and context parallelism size 4, per-rank KV memory falls from 0.77 GB to 0.20 GB. The feature requires prefill context parallelism with interleaved strategy.

UnifiedRadixTree becomes the default for sliding-window attention, Mamba, and DSA models. Replay SSM and Mamba int8 checkpoints sync onto it, and a cache hit now resets only the state it used.

New model support and kernel work

Inkling is a 975B-parameter multimodal MoE with a 1M-token context that mixes sliding-window, full, and Mamba2 linear attention. It adds an NVFP4 MoE, optional vision and audio towers, and native multi-token prediction. On Blackwell the project reports up to 71.7k tokens per second input and 171.0 tokens per second per-user decode. Verification covered Blackwell TP4 and TP8, H200, and AMD MI350X and MI355X.

Other additions include LongCat 2.0 FP8, JetBrains Mellum v2, Pi0.5 for vision-language-action, and diffusion support for LongLive 2.0. MiniMax-M3 completed its four-part landing but its cookbook still points at a development image.

Linear attention on Blackwell SM100 gets a new recurrent KDA decode kernel that runs at 29.6 microseconds versus 36.8 microseconds for Triton at batch size 64. The full decode path reaches parity at batch size 128 and 1.35x at batch size 256, but is slower below that.

Breaking changes and known issues

The experimental QServe W4A8 and FBGEMM FP8 quantization paths are removed. The FP4 GEMM backend option for CUTLASS is gone along with in-tree NVFP4 JIT kernels, so NVFP4 GEMM now requires FlashInfer. The auto backend picks FlashInfer CuteDSL on SM100 and FlashInfer CUTLASS on SM120. CUTLASS FP8 blockwise is deleted for SM90 and SM100, with SM120 moved to JIT.

Chunked input-logprob processing is now on by default to cap peak memory. FA3 sparse mask kernels are off by default. Two flags were renamed without deprecated aliases: the DeepEP waterfill option becomes enable-waterfill, and optimistic prefill retries becomes optimistic prefill attempts. The diffusion post-training rollout endpoint now returns msgpack instead of JSON, so consumers must upgrade in lockstep.

A known issue involves temperature-0 nondeterminism under DP attention with breakable prefill CUDA graph on the DSV4-Flash FP4 recipe, where an idle-rank dummy extend perturbs real requests' logits. The guarding determinism test is disabled as a stopgap. A flashinfer 0.6.15 bump was reverted, so the release pins 0.6.14. GB300 CI jobs were temporarily disabled for runner availability, so coverage rests on manual validation.