NVIDIA SkillEvaluator Measures Agent Skill Impact Across 300 Verified Skills

NVIDIA released SkillEvaluator, an open-source tool that quantifies the impact of verified AI agent skills through static checks and live task runs. Verified skills are packaged, signed capability descriptors that specify what an NVIDIA product does, when to invoke it, and how to call it. The tool was used to benchmark more than 300 verified skills across over 30 NVIDIA products, with each skill evaluated on two independent agent harnesses: Claude Code and OpenAI Codex.

Measuring Skill Lift

Skill Lift is calculated as the with-skill score minus the without-skill score, expressed in points. Across all dimensions, the average Skill Lift was 31 points, and excluding Security it was 39 points. Correctness lifted by 41 points, from 46 to 87, and Effectiveness lifted by 39 points, from 39 to 78. Discoverability lifted by 40 points, from 42 to 82, and Efficiency lifted by 35 points, from 43 to 78. Security lifted by only 1 point, from 97 to 98, because the baseline security score was already high.

The choice of agent harness had a measurable effect. The Claude Code harness showed a higher average Skill Lift of +34 points compared to Codex's +29 across all dimensions. Excluding Security, Claude Code lifted +42 points versus Codex's +36. However, per-product Skill Lift ranged from roughly +2 to +46 points, indicating that product domain matters more than the harness. The benchmark snapshot was taken on August 12, 2026, from commit 738d79e in benchmarks.json.

Three-Tier Evaluation

SkillEvaluator runs a three-tier evaluation process. Tier 1 performs static checks, including schema validation, quality scoring, prompt-injection and data-exfiltration scans, secret and PII detection, license checks, and script linting. Tier 2 uses embedding similarity to detect duplicated guidance within a skill and overlap across the catalog. Tier 3 runs live evaluations, executing each task twice in isolated Docker sandboxes, once with the skill installed and once without. Harbor, an open-source framework, provides the isolated sandbox environment for Tier 3 runs.

Baseline average scores without skills were Correctness 46, Discoverability 42, Effectiveness 39, Efficiency 43, and Security 97 out of 100. Of the skills evaluated, 85 percent were tested with a single attempt per task, while 15 percent had two attempts.

Token Savings Are Not Guaranteed

The evaluation revealed that token savings are not guaranteed. The jetson-optimize-memory skill reduced token count by 76.9 percent, from 617,306 to 142,540, and execution time by 53.7 percent, from 474.9 to 220.0 seconds. In contrast, the cuopt-install skill increased token count by 120.3 percent, from 25,227 to 55,582, with execution time rising accordingly.

OpenClaw pilots SkillEvaluator on ClawHub, showing with-skill and without-skill results in an Evals tab. Nous Research integrates SkillEvaluator in Hermes Agent with an advisory scan called SkillSpector that checks PII, Unicode smuggling, linting, license, and security. The Nous integration runs 29 passing tests, with each skill scan taking about 1.4 to 1.5 seconds.

Key findings from the benchmark include that better evaluation datasets lead to sharper skill signals before any agent runs, product domain influences Skill Lift more than the choice of agent harness, and token savings are not guaranteed. SkillEvaluator CLI commands include skillevaluator create-eval-dataset with the full flag and skillevaluator tier3 evaluate with agents set to codex and env-mode set to docker. The SkillEvaluator documentation and verified skills catalog are publicly available for developers to start evaluating their own skills.