NVIDIA RoboLab Benchmarks Robot Policies With Capability-Specific Tasks

NVIDIA introduced RoboLab, a simulation benchmarking platform for evaluating general-purpose robot policies across different embodiments, on July 11, 2026. The platform addresses three problems in existing benchmarks: visual domain overlap between training and evaluation, benchmark saturation from static task sets, and the lack of diagnostic detail in binary success rates.

What RoboLab changes

RoboLab separates the visual source used for training from the one used for evaluation. Most benchmarks draw training and evaluation data from the same simulated environment, so strong performance can mean the model memorized the setup rather than generalized. Real2sim approaches reconstruct photorealistic scenes from real images using Gaussian Splatting, but per-scene setup can take over an hour, making large-scale testing impractical. RoboLab instead compiles the same scenes and tasks against whichever robot a user brings, keeping the benchmark robot- and policy-agnostic.

The platform also generates new tasks quickly. Users place objects in a scene, specify a language instruction, and the process takes minutes. Agent skills let a coding agent create novel tasks directly in a user's workflow, so benchmarks can retire outdated tasks as models improve. This counters saturation, where models max out scores on fixed task sets and every system reports over 90% success, making the numbers meaningless.

RoboLab tags each task with specific capabilities. Visual competency tests recognition of color, size, and semantic category. Procedural competency covers action-oriented reasoning like stacking and reorienting. Relational competency probes spatial and linguistic logic, including conjunctions, counting, and relative positions. The initial benchmark, RoboLab-120, contains 120 human-curated tabletop pick-and-place tasks, each tagged with the capabilities it requires.

Metrics beyond success rate

RoboLab uses three additional evaluation tools alongside success rate. Graded task scores give partial credit for completing subtasks within a multi-step instruction, so a robot that grasps the right object but misses the drop target is not scored the same as one that does nothing. Trajectory quality measures motion efficiency via path length and SPARC, a spectral arc-length metric that captures smoothness through the Fourier spectrum of velocity. Speed of execution measures end effector velocity, reflecting the human perception that faster motion is preferred.

Failure event logging tracks wrong-object grasps, dropped objects, and gripper collisions, pinpointing where execution derails. The platform also includes a dashboard for analyzing these events.

Statistical trustworthiness is handled with the Clopper-Pearson method, an exact binomial confidence interval. For an observed 90% success rate with 70 rollouts, a 95% confidence interval spans 15.4 percentage points, from 80.5% to 95.9%. With 1,030 rollouts, the interval narrows to a 2 percentage-point band, from 88.0% to 91.8%. Most published benchmarks do not run enough rollouts for statistically significant comparisons between two policies.

Sensitivity analysis uses Neural Posterior Estimation to quantify which environmental variables most affect performance, replacing guesswork with statistical attribution. Testing across language, scene, and task-horizon complexity shows current policies remain brittle to vague instructions and struggle with long-horizon tasks exceeding four subtasks.

NVIDIA plans to productize RoboLab in Isaac Lab-Arena. The paper and code are available for researchers who want to run evaluations or contribute new tasks.