NVIDIA Open-Sources SkillEvaluator to Test Whether AI Agent Skills Actually Work

Takeaways

• NVIDIA open-sourced SkillEvaluator , an Apache-2.0 framework for evaluating agent skills end to end.
• Three independent tiers: static validation, embedding-based deduplication, and live sandboxed agent runs.
• Tier 3 measures Skill Lift, the behavioral delta between agent runs with and without the skill.
• Scores on five dimensions plus pass@k reliability, with CLI, JSON, HTML, and Markdown reports.
• Ships GitHub Actions and GitLab CI templates for merge-gating skills on pull requests.
• Works keyless for quality checks; LLM rubric and Tier 3 need a provider and Docker sandbox.