A Claude Code plugin for evaluating the skills and agents you build. Stop guessing whether your skill works — find what's actually breaking, scaffold the cheap deterministic checks, grade the subjective stuff with an LLM judge you've proven you can trust, generate test data when you have none, and run the whole suite on demand.
Built for anyone shipping Claude Code skills/agents (and happily dogfooded on real ones). Grounded in the eval patterns from awesome-evals.
The path
"Evals" gets thrown around to mean ten different things, and most people skip the one that matters: looking at what their thing actually does wrong. This plugin encodes the high-leverage path, cheapest rung first:
- Read real traces and label what broke — before inventing any metric →
error-analysis - Scaffold deterministic checks for the failures code can catch — free, no drift →
add-assertions - Grade the subjective failures with an LLM judge — validated against human labels so it isn't a rubber stamp →
build-judge - Generate a grounded eval set when you have no traffic yet →
synth-data - Measure capability vs reliability honestly →
passk - Prove the skill earns its context window — with/without A/B eval →
skill-lift - Run the whole suite and get a report →
eval-runner(agent)
Skills
error-analysis
Find what's actually breaking before building metrics. Walks the open-coding → axial-coding → prioritize loop: sample 20–100 traces, label the first failure in each, cluster into 4–8 named categories, and rank by frequency × severity. Outputs a failure taxonomy and decides, per failure, whether it needs a cheap code assertion or an LLM judge.
Triggers: "do error analysis", "what's breaking in my skill", "open code my traces", "build a failure taxonomy".
add-assertions
Turn the deterministic failures from error-analysis into runnable code checks — the cheapest eval rung, no model call, zero drift. Ships a catalogue of patterns (valid JSON, required fields, no unsubstituted placeholders, enum/range, tool actually fired, side effect occurred, no collateral damage) plus a runnable JSONL trace runner that exits non-zero for CI. Core rule: assert on the state of the world, not what the transcript claims — models cheerfully report success they didn't achieve.
Triggers: "add assertions", "scaffold assertions", "deterministic eval checks", "assert my agent output".
build-judge
Build a binary LLM-as-judge for one subjective dimension (tone, faithfulness, instruction-following) and validate it properly. The key move: score the judge against human labels using true-positive rate and true-negative rate separately — because on imbalanced data a judge can hit 90% accuracy while catching zero real failures. Includes a stdlib-only scoring script that gates on both rates and exits non-zero for CI.
Triggers: "build an llm judge", "validate my judge", "llm as judge", "check if my judge agrees with humans".
python3 skills/build-judge/scripts/score.py labels.csv --min-tpr 0.85 --min-tnr 0.85
synth-data
Generate a realistic, diverse eval set when you have no production traffic. Define dimensions (features × scenarios × personas), take the cartesian product, generate one input per tuple (never bulk — that collapses into a few repetitive shapes), ground every input in real system state, and keep only the cases whose assertions actually fire. Output is JSONL that feeds add-assertions and the eval runner.
Triggers: "synthetic eval data", "generate test cases", "no production traffic", "bootstrap evals".
passk
Measure capability versus reliability the right way. pass@k (at least one of k samples succeeds) and pass^k (all k succeed), with the unbiased estimator — not the naive 1-(1-p)^k. The two move in opposite directions: a model that succeeds 75% per attempt is pass@10 ≈ 1.0 but pass^10 ≈ 5.6%. Reliability is the brutal, honest number for customer-facing agents.
Triggers: "pass@k", "pass^k", "capability vs reliability", "unbiased pass@k estimator".
python3 skills/passk/scripts/passk.py results.csv # sweeps k = 1,2,5,10skill-lift
Measure whether a SKILL.md actually lifts agent performance. Runs the same task set under two conditions — with and without the skill loaded, same model, same grader — and computes the pass-rate delta. Flags tasks where the skill made things worse (regressions), which are usually the most actionable finding. Supports version A/B comparison (--baseline v1 --treatment v2) and aggregated or per-run input formats.
Triggers: "measure skill lift", "does my skill help", "with/without eval", "A/B test my skill", "skill lift", "benchmark my skill", "does the skill earn its context window".
python3 skills/skill-lift/scripts/lift.py results.csv python3 skills/skill-lift/scripts/lift.py results.csv --baseline v1 --treatment v2
Agents
eval-runner
Orchestrates a full eval run end to end: locate the dataset + target + checks, produce or ingest outputs, run the deterministic assertions and any LLM judges, aggregate overall and per-category pass rates, call score.py for judge TPR/TNR and passk.py for pass@k/pass^k where the data supports it, and emit a concise markdown report with a prioritized fix list tied back to the failure taxonomy. Honest about gaps — no labels means it won't pretend to validate a judge; a single sample per task means no reliability number.
Invoke: "run my evals", "score my skill against the dataset", "produce an eval report".
Install
Local (for development / dogfooding):
claude --plugin-dir /path/to/skill-evals
Then just talk to it — the skills auto-trigger on the phrases above, or invoke explicitly via /skill-evals:error-analysis, /skill-evals:add-assertions, /skill-evals:build-judge, /skill-evals:synth-data, /skill-evals:passk. The eval-runner agent runs the whole suite.
Roadmap (post-0.3)
ci-gating— regression datasets and deploy gates as a first-class skilloutcome-evals— environment-state grading helpers (diff the world, not the transcript)trajectory-evals— tool-call path checking (deterministic match + LLM-judge-of-trajectory)contamination-resistant— date-stamped tasks to separate memorization from reasoning
License
MIT