ParallelKernelBench: Frontier LLMs struggle with multi-GPU CUDA kernels

Together AI released ParallelKernelBench (PKB), a benchmark that tests how well frontier large language models can write custom CUDA kernels for multi-GPU workloads. PKB covers 87 real-world problems, each replacing a standard PyTorch plus NCCL reference implementation with a custom kernel that uses symmetric memory across GPUs. The problems span sharding types including tensor, context, data, expert, sequence, FSDP/ZeRO, vocab, graph, geometric/spatial, pipeline, and collective primitives, drawn from systems like Megatron-LM, DeepSpeed, DeepEP, TensorRT-LLM, and NeMo-RL, plus distributed FFTs and Gaussian splatting.

Results and gaps

In zero-shot evaluation, the best model, GPT-5.5, solved 28 of 87 problems correctly. Of those, 22 outperformed the PyTorch plus NCCL baseline. Sampling three attempts per problem raised GPT-5.5's correct solutions to 36 and faster-than-baseline solutions to 27, giving a fast-1@3 metric of 31 percent. Other frontier models, including Claude Opus 4.7, Gemini 3 Pro, GLM-5.1, GLM-5.2, and DeepSeek V4 Pro, scored lower on both pass@k and fast-1 metrics.

Successes clustered in collective primitives, tensor-parallel GEMMs, and Ulysses-style context parallelism. Failure modes varied by model strength: weaker models often failed to compile, while stronger models compiled but produced incorrect results or deadlocks. Generated kernels mostly relied on copy engines or SM load/store operations, with advanced mechanisms like TMA and NVLS rarely chosen.

Agentic refinement and wins

An agentic evaluation loop, where the model iterates with compiler, tests, and benchmarks, modestly improved Gemini 3 Pro from 24 to 35 correct solutions and from 24 to 26 faster-than-baseline kernels. Performance gains plateaued after roughly 20 refinement steps.

A generated GEMM plus All-Gather kernel ran in 87.9 microseconds versus 320.6 microseconds for the PyTorch plus NCCL reference, though still above the theoretical 4.99 microsecond roofline. The benchmark's communication-aware roofline analysis shows most PKB problems are NVLink-bound, and baseline implementations run far below hardware ceilings.

Three net-new kernels beat any publicly available implementation, including a NeMo-RL GRPO vocab-parallel log-prob kernel. That kernel was verified correct on 4 H100 GPUs over 100 random runs, using ThunderKittens 2.0 benchmarking methodology. Speedup was measured across sequence lengths 1024 to 16384 with 4 ranks, batch 1, vocab 32000 (8000 per rank), top-k 10, and top-p 0.9.

Implications for inference engineers

For those serving models, PKB highlights that current LLMs cannot reliably produce production-grade multi-GPU kernels, especially for complex parallelism patterns. The benchmark is hardware-agnostic, using standard PyTorch plus NCCL references, so it can evolve with future GPU generations. The modest gains from agentic loops and the plateau after 20 steps suggest that while iterative refinement helps, fundamental kernel-writing ability remains a bottleneck. The few net-new kernels, however, show that LLMs can occasionally discover implementations that beat existing hand-tuned code, offering a glimpse of potential for automated kernel optimization in specialized workloads.