My skill evaluator gave a Grade C to one of the best skills ever written.

TL;DR — My skill evaluator, built on Anthropic’s 12-criteria framework, gave Matt Pocock’s writing-great-skills a 58/100. Grade C. For a skill written by the guy who wrote the manual. The rubric was measuring feature presence and punishing correct decisions. I merged it with Pocock’s framework into a v2 that scores behavior instead of checkboxes: same skill, re-run by a context-free agent, 82/100, Grade A.

The skill wasn’t the problem. My rubric was.

I maintain a skill that evaluates other agent skills. It scores a SKILL.md against twelve criteria from Anthropic’s “Lessons from building Claude Code” post, runs a weighted average, and prints a letter grade. Solid, I thought. It even shipped with its own quality checklist.

Then I ran the transcript of Matt Pocock’s AI Engineer talk, “The Missing Manual: How to Write Great Skills,” through a Claude session, side by side with his writing-great-skills skill, and asked one question: what does his framework measure that ours doesn’t?

Almost everything that matters, it turns out.

The Anthropic criteria mostly check for the presence of features. Does the skill have a gotchas section? Scripts? A memory mechanism? Progressive disclosure across multiple files? Those are decent proxies. But they’re proxies.

Pocock’s framework asks about behavior instead. Four axes:

- Trigger — did you choose deliberately between model-invoked (the description costs context on every single turn) and user-invoked (you become the index that has to remember it exists)?

- Structure — is material every branch needs inline, and material only some branches need behind a pointer whose wording says when to follow it?

- Steering — do you use leading words, compact terms with strong priors (”vertical slice”) that you can literally watch reappear in the agent’s reasoning traces? Does each step end on a completion criterion the agent can check?

- Pruning — the deletion test. Remove a sentence; if behavior doesn’t change, it was a no-op and you were paying tokens to say nothing.

One framework counts what a skill has. The other tests what each line does.

Here’s where it got uncomfortable. I scored Pocock’s own writing-great-skills with our v1 rubric: 58/100. Grade C. “Functional but significant gaps.”

This is a skill of 83 dense lines plus a glossary, written by the guy who wrote “the manual”. The C came from:

- 0/100 on memory, scripts, and hooks — features a pure-reference skill correctly doesn’t have

- 20/100 on its description, because it carries no trigger phrases. It’s deliberately user-invoked; a trigger list would be wrong there

- 55/100 on disclosure, because our criterion counted directories (”has references/?”) while his glossary sits behind one precisely worded pointer, which is exemplary disclosure

Every one of those penalties punished a correct decision.

I merged the two frameworks into v2: 18 criteria across Pocock’s four axes, 14 core plus 4 that only apply when the skill’s category calls for them — otherwise they’re N/A and drop out of the average entirely, numerator and denominator. Description quality is now judged against the invocation mode the skill actually chose. And the “Top 3 Improvements” list became failure-mode diagnosis: premature completion, sediment, sprawl, weak steering, no-ops — each with file:line evidence, a root cause, and the specific defense.

Then came the real test: a second subagent, completely fresh, zero context from the first session, got two file paths and one instruction — follow the evaluator.

v2 gave Pocock’s skill 82/100. Grade A. Same skill, same files, 24 points apart. The ruler had been broken, not the thing it measured.

But the part I keep coming back to: v2 wasn’t a rubber stamp. The context-free agent found a real duplication — and it was ours. While building v2 I’d condensed Pocock’s glossary into a reference file, and the evaluator flagged that copy, then nailed the root cause: his skill is user-invoked, so no other skill can reach it, which forces consumers to copy the definitions. That’s a sharper finding than anything v1’s “top improvements” format ever produced.

Evaluate your evaluator against something you already know is excellent. If the grade comes back mediocre, suspect the rubric before you suspect the skill.

And when you write rubrics for agents, ask what each criterion actually measures. “Has multiple files” is easy to check and nearly meaningless. “Does deleting this sentence change the agent’s behavior” is harder to check — and it’s the whole game.

What does your rubric actually measure? If you’ve built evals for agent skills (or got burned by one), I’d genuinely like to hear how you score them — leave a comment.

Deixe um comentário

Check my Skills repo: https://github.com/fabricioctelles/skills