NVIDIA's LLM Co-Design Guide: Dimensioning Rules for Blackwell GPUs

NVIDIA published a practical guide on July 10, 2026, explaining how model designers can shape LLM architecture to run better on modern GPUs. The post, written by Ritika Borkar, Nidhi Bhatia, Bhargava Gopireddy, Nick Comly, Brian Pharris, Julien Demouth, and Bita Darvish Rouhani, focuses on throughput and interactivity while holding accuracy fixed. The core argument is that models aligned with hardware not only run faster but also scale better and cost less.

The guide frames deployment along two axes: workload regime, from short to long context, and service goal, from throughput-oriented to latency-oriented. Each quadrant demands different optimizations. Long-context, throughput-oriented serving spends most time in attention, while latency-oriented serving adds model parallelism to shorten attention and feed-forward networks, at the cost of communication overhead. Short-context, throughput-oriented serving splits time more evenly and benefits from parallelism like expert parallelism at scale. Amdahl's law applies: if attention takes 77% of runtime, tuning feed-forward layers yields marginal gains.

Dimensioning rules for linear layers

The key design choice is the aspect ratio between model width and layer count. Width is set by hidden dimension H and intermediate projection dimension H prime in MLP layers. Together with layer count L, these determine how cleanly the model maps onto parallelism strategies and scales across GPUs.

Arithmetic intensity, defined as compute operations per byte of memory moved, determines whether a workload is memory-bound or compute-bound. Low arithmetic intensity is capped by memory bandwidth; high arithmetic intensity is capped by peak compute throughput. For maximum throughput, the goal is to push workloads into the compute-bound region. Latency-sensitive decoding runs at low concurrency and is memory-bound, so reducing memory-access time lowers response latency.

A concrete example on GB300 illustrates the point. For FFN-2 with H prime of 512 and H of 8192, using 4-bit inputs and 8-bit outputs, the layer stays memory-bound at every token count. Even at 16,384 tokens, memory time exceeds math time. The write cost dominates because outputs are FP8 while inputs are FP4, and the small reduction dimension keeps the GEMM memory-bound.

NVIDIA's first guideline: for a fixed-parameter model, favor near-square weight matrices and avoid making either the projection or reduction dimension too small. But size alone is not enough. Dimensions must also map cleanly onto the underlying tiling geometry. Poor alignment causes tile quantization, reducing throughput even with high arithmetic intensity.

Tile alignment and GPU execution

GPUs execute GEMMs by splitting the output matrix into tiles, each computed by a streaming multiprocessor. On recent GPUs, SMs can cooperate on larger tiles: clusterMMA lets two neighboring SMs join forces, and a Cooperative Grid Array lets a whole group work together as one cluster. Cooperation improves data reuse but enlarges the effective tile, so dimensions must be multiples of larger values to stay aligned. When a dimension is not a multiple of the effective tile, edge tiles are only partially filled yet still run a full tile's worth of compute, wasting cycles.

Sweeping GEMM-N in fine-grained steps produces local throughput maxima when N is a multiple of 256 or 512, depending on the cluster configuration. The guide recommends aligning dimensions to multiples of 128, preferably 256 or 512, to avoid tile quantization and keep Tensor Cores saturated on Blackwell GPUs.

Wider, shallower models and quantization

Wider, shallower transformers achieve higher arithmetic intensity and lower latency than deeper models at the same parameter budget. This benefits both throughput and interactivity. The guide also covers NVFP4 quantization, which closely matches FP8 accuracy on benchmarks such as DeepSeek-R1 while delivering 4-bit compute speed. NVIDIA Model Optimizer supports post-training quantization and quantization-aware training to NVFP4.

For Mixture-of-Experts models, scaling expert parallelism across many GPUs raises global concurrency and aggregate memory bandwidth. TensorRT-LLM's Wide-EP feature addresses all-to-all communication and load imbalance. Regular, repeatable layer patterns enable balanced Chunked Pipeline Parallelism stages for prefill, while hybrid strategies such as Helix Parallelism decouple attention and FFN parallelization to meet latency-oriented service goals.

The post is the first in a series targeting different dimensions of hardware-aware design. Subsequent chapters will address avoiding compute bottlenecks, frictionless deployment at datacenter scale, matching design to use case, and scaling in practice.