NVIDIA Details Full-Stack Energy Efficiency Levers for AI Factories

Power can account for 40% of operating expenses in an AI factory, and most sites run under a fixed power cap from the regional provider. NVIDIA says performance per watt is therefore the metric that directly sets token cost and profitability. The company describes a set of levers across hardware, precision, scheduling, and grid integration that operators can pull to raise tokens per watt without increasing total site power.

System and precision levers

NVIDIA states that across six architecture generations it has improved inference throughput per megawatt by 1,000,000x. The GB200 NVL72 rack-scale system uses direct-to-chip liquid cooling and in-rack power smoothing to flatten peak current spikes, which lets operators deploy more GPUs within the same power budget. The DSX platform adds dynamic power allocation, real-time telemetry, and rack-level controls that recover stranded power.

Precision choice is another lever. Higher-precision calculations are slower and consume more energy, while narrow formats like NVFP4 deliver higher throughput at equivalent accuracy to FP8. NVIDIA Dynamo and TensorRT-LLM translate those gains into real-world inference performance by boosting throughput and scaling reasoning models across GPU infrastructure. For model architecture, mixture-of-experts designs are typically more energy efficient per unit of intelligence than dense models with similar total parameters, because only a subset of experts activates per token. NVIDIA cites DeepSeek-R1 as an example of a large-parameter MoE model that achieves higher task performance at similar or lower per-token compute cost than dense predecessors.

Training energy optimization

During large-model training, pushing for maximum iteration speed consumes large amounts of energy, and imbalanced GPU workloads leave many GPUs idle while waiting for stragglers to sync. Researchers from the ML.ENERGY Initiative at the University of Michigan showed that tuning individual GPU processing speeds reduces energy bloat. GPUs on the critical path run at maximum speed, while those with less work are intentionally slowed down. This minimizes idle time, lowers energy use on slower GPUs, and keeps end-to-end training time unchanged.

NVIDIA Megatron-LM integrates fine-grained kernel and phase-level energy profiling to identify compute, memory, communication, and power-limited regions. Energy-aware scheduling and GPU frequency or power-cap tuning align with the true critical path of training iterations. The work targets up to roughly 25% energy savings at similar iteration step time, freeing power for additional training runs or inference on the same infrastructure.

DSX platform layers

DSX MaxLPS operates inside the AI factory and includes 45°C liquid cooling to improve power usage effectiveness, dynamic power allocation that reallocates power within defined budgets, and advanced techniques like power steering and NVIDIA Dynamo orchestration for inter-rack optimization. DSX Flex connects the factory to grid signals and external energy sources for end-to-end power optimization. The ML.ENERGY leaderboard provides benchmarks that can be tied into telemetry-driven operations, helping operators understand where power is used, where it is stranded, and how much additional compute fits within a fixed site envelope.