Evals Skills for Coding Agents
Today, Shreya Shankar and I are publishing evals skills, a set of skills for AI product evals1.
Eval tools often get in the way. They nudge you toward generic off-the-shelf metrics and fully automated evals before you’ve looked at your data. These skills help you avoid common mistakes we’ve seen helping 50+ companies and teaching students in our AI Evals course.
Why skills for evals
There are many easily avoidable footguns in evals. These skills help you avoid them.
start is the entry point. It looks at your situation and routes you to the right skill. Most of the time it will send you to one of these two:
- eval-audit, if you already have an eval pipeline. It inspects your setup and recommends next steps. 2
- error-discovery, if you have traces but haven’t analyzed them yet. It builds a customized annotation interface and helps you sample traces intelligently. Shreya does a live walkthrough of using this skill here.
The skills
Install the skills:
npx skills add https://github.com/ai-evals-course/evals-skillsThen give your agent this prompt:
Run the start skill from the evals plugin and follow the skill it picks. If it picks eval-audit, investigate each diagnostic area using a separate subagent in parallel, then synthesize the findings into a single report.
If you’re experienced with evals, skip the router and pick the skill you need:
| Skill | What it does |
|---|---|
| start | Entry point. Routes to the skill that matches your situation |
| eval-audit | Audit an eval pipeline and surface problems with prioritized severity |
| error-discovery | Build a review app, select diverse samples, and organize your notes into failure modes |
| generate-synthetic-data | Create diverse synthetic test inputs using dimension-based tuple generation |
| write-judge-prompt | Design LLM-as-Judge evaluators for subjective quality criteria |
| validate-evaluator | Calibrate LLM judges against human labels using data splits, TPR/TNR, and bias correction |
| evaluate-rag | Evaluate retrieval and generation quality in RAG pipelines |
| build-review-interface | Build custom annotation interfaces for human trace review |
These skills are only a starting point. To make them better, tune them to be more specific to your data and domain.
The repo is ai-evals-course/evals-skills.
You can find me on X or email me through my newsletter.
Footnotes
Not foundation model benchmarks like MMLU or HELM that measure general LLM capabilities. Product evals measure whether your pipeline works on your task with your data. If you aren’t familiar with product-specific AI evals, check out my AI Evals FAQ.↩︎
The audit isn’t a complete solution, but it will catch common problems.↩︎