NVIDIA's Generative Recommenders Cut Latency and Boost Throughput on H100

NVIDIA has published a detailed look at how generative recommender systems are being adapted for large-scale production, with a focus on training and inference optimizations for two model families: HSTU and Semantic ID. The company's recsys-examples repository now includes production-ready code for both, covering the full pipeline from training to serving.

Training and Inference Optimizations

For HSTU training, NVIDIA uses Megatron-Core to enable data, tensor, sequence, and pipeline parallelism. On two DGX H100 nodes, this approach improves Model FLOP Utilization to 31.40% from 7.65%, a significant jump in hardware efficiency. The HSTU architecture itself replaces the standard softmax attention with SiLU weighting, adds relative bias, and applies elementwise gating before projection, which contributes to its efficiency.

On the inference side, HSTU with the PyTorch AOTInductor backend, integrated with Triton Inference Server, delivers a 1.14x to 1.28x speedup over a Python backend when no KV cache is used. When a KV cache is employed, the speedup grows to 2.20x to 2.38x in an ideal all-GPU cache-hit scenario. This is complemented by nv-embedding-cache, a hierarchical caching system that spans GPU, CPU, and remote memory with lockless invalidation, and offers drop-in NVEmbedding and NVEmbeddingBag modules for PyTorch.

DynamicEmb for Long-Tail Catalogs

A key challenge in recommender systems is handling long-tail item catalogs, where static embedding tables become impractical. NVIDIA's DynamicEmb addresses this by replacing static tables with a GPU-optimized scored hash table that allocates rows on demand across HBM and host memory. It uses admission control and score-based eviction to prioritize important IDs, making long-tail catalogs tractable at scale. DynamicEmb is implemented as a TorchRec backend, sharded row-wise via the EmbeddingBagCollection and EmbeddingCollection APIs, and uses fused CUDA kernels for SUM, MEAN, and sequence pooling lookups and gradient reduction.

Semantic ID Inference Gains

For Semantic ID models, NVIDIA has developed specialized inference that separates shared context KV, per-beam decode KV, and beam ancestry. This design is tailored to the workload characteristics of Semantic ID, which typically involves thousands of history tokens, decoding only 2-3 tokens, and beam widths of 128-256. The result is a 2.14x to 2.27x reduction in offline latency compared to SGLang beam search on H100 GPUs, and a 1.85x increase in online throughput. These figures are NVIDIA's own benchmarks, but they indicate substantial performance improvements for serving generative recommenders at scale.

For inference engineers, the practical takeaways are clear: HSTU with AOTInductor and KV caching can nearly double throughput in cache-hit scenarios, while DynamicEmb offers a scalable solution for embedding tables that would otherwise exceed GPU memory. The Semantic ID optimizations show that specialized inference kernels can outperform general-purpose beam search by a wide margin, making generative recommenders a viable option for production environments.