NVIDIA NeMo Gym cuts Siemens agent turns 66% in chip design eval

Siemens EDA used NVIDIA NeMo Gym to evaluate and improve the Solido Characterization Suite agent for standard-cell characterization, raising the composite verification score from approximately 55% to 95% on a representative benchmark. The optimization also cut agent turns from 155 to 53, a reduction of about 66%, and output tokens from roughly 81K to 29K, a reduction of about 64%. The results come from a development benchmark using the open source GF180MCU PDK, with 229 standard cells across 36 PVT corners, though each evaluation run was restricted to two cells across those corners for tuning.

Verifying final artifacts

NeMo Gym, an open library for evaluating and improving models and agents, provides an agentic evaluation environment that diagnoses failure modes and tracks error propagation across multistage workflows. The Siemens team built a custom environment that defines the task, tools, state, and verifier for each rollout, recording thinking traces, messages, and tool actions. The verifier checks final Liberty files against known references using five metrics: coverage of expected cells and PVT corners, validity of the file format, relative validation against pre-existing failures, accuracy of timing and power values within tolerance, and integrity to ensure the workflow produced complete results rather than reusing or truncating existing values. The reward score is a product of binary gates for coverage, validity, relative validation, and integrity, multiplied by the fraction of output values within tolerance. If any gate fails, the reward is zero.

This end-to-end verification exposed failures that intermediate checks missed, including missing cells and operating conditions, incorrect slew thresholds, and timing or power values outside tolerance. The team ran approximately 25 rollouts using Claude Opus 4.8 in Claude Code as the fixed model and agent harness, with each run taking about 1.5 to 2 hours. They refined the agent's skills and Model Context Protocol tool descriptions, simplifying overlapping file and shell interfaces and defining tool inputs with explicit JSON schemas to improve tool selection.

Applying the method elsewhere

The same evaluation approach applies to engineering agents in EDA, CAE, and other domains. The key steps are evaluating final engineering artifacts rather than individual tool calls, writing a domain-specific verifier that detects missing, truncated, reformatted, or reused results, and collecting large numbers of rollouts to target reliability improvements through optimized skills and MCP interfaces. NVIDIA provides a NeMo Gym Quickstart guide for initial evaluations, and the Siemens work demonstrates the method on a representative task: define a valid final artifact, build a verifier, inspect rollout failures, refine skills and tool descriptions, and rerun under consistent conditions.