NVIDIA AIPerf Replaces GenAI-Perf for High-Concurrency LLM Benchmarking

NVIDIA introduced AIPerf as the successor to GenAI-Perf, a ground-up rewrite that addresses client-side bottlenecks in high-concurrency LLM benchmarking. The tool uses a multiprocessed system where worker processes generate load, separate record-processor services handle results, and coordination happens over ZMQ. This architecture prevents the benchmarking client from becoming the limiting factor when testing servers under real concurrency or request rates.

What AIPerf changes

AIPerf does not run on top of Perf Analyzer the way GenAI-Perf did. The clean architectural break is the reason the tool can scale. For teams porting existing workflows, a migration guide covers the key differences between the old and new tools.

The tool supports more than 15 endpoint types, including chat, responses, NIM rankings, and image generation. It also works with public datasets like ShareGPT and trace replay formats from Mooncake, Baseten, and WEKA AgentX. Engineers can run quick synthetic smoke tests or replay captured production traffic without switching tools.

Load shaping is configurable beyond simple volume. AIPerf supports constant, Poisson, and gamma arrival patterns with tunable burstiness, gradual ramping for concurrency and request rate, and synthetic distributions including vLLM and SGLang range-ratio for variable input and output sequence lengths.

Running a first benchmark

The walkthrough uses Qwen3-0.6B served through vLLM, a model small enough to run on a single GPU and fast enough to iterate without waiting. The goal is to establish the measurement loop, not to benchmark that specific model.

A static benchmark pins input and output tokens to exactly 128 per request using standard deviation flags set to zero. Two extra flags matter here: min_tokens and ignore_eos tell the model to actually emit 128 tokens rather than stopping early. Without them, the output token count is a suggestion and throughput numbers end up lower and not reproducible across runs. Streaming is required to measure time to first token and inter-token latency, since without it the server batches the full response before sending anything.

Reading the metrics

AIPerf reports four core metrics: time to first token (TTFT), inter-token latency (ITL), request latency, and output token throughput. Each comes with percentile breakdowns at p25, p50, p75, p90, p95, and p99, plus minimums, maximums, averages, and standard deviations. These breakdowns matter because a server with a healthy mean TTFT and an outlier p99 can look fine in aggregate and fail in production.

When DCGM or pynvml is available, AIPerf also pulls GPU power draw, utilization, and memory consumption into the same run output. Correlating a latency spike with memory pressure does not require a separate profiling session.

Configuring realistic traffic

A Poisson arrival pattern with a request rate of 10 means requests arrive at an average of 10 per second with inter-arrival times drawn from an exponential distribution. The server experiences bursts and gaps rather than a single user stream, which is what queuing looks like under real traffic. Input token standard deviation of 128 introduces variance around a 512-token mean, producing a mix of short and long prompts. Output token standard deviation of 32 adds variance on the output side, releasing the constraint that pinned outputs to exactly 128 tokens in the static baseline.

A random seed makes the Poisson timing and synthetic length draws reproducible, so rerunning the command produces the same sequence of requests. The resulting distributions are noticeably wider than the static baseline, which is expected when more requests compete for GPU access and prefill lengths vary per request. The tool writes full results to CSV and JSON in addition to printing a metrics table to the console.