The concise version of my post, “AI writes both the tests and the code, so everything passes but nothing is actually validated.”

This article is the follow I mentioned, how I built the skill that replaced that loop, step by step, so you can built similar skills.

The Problem I Was Solving

I had 20 PRs open in parallel across projects. Every single one had tests written by Claude Code. Every single one passed.

The issue wasn't the test runner. The issue was intent.

AI was guessing what to test, guessing how to implement, then confirming its own guesses.

I needed a way to inject my understanding back into that loop without going back to writing tests by hand.

The Meta Skill

Link to skill

I have a skill called create-skill. It’s a meta skill, a skill that builds other skills.

It enforces structure. Every skill needs a SKILL.md under 100 lines. A description that tells the agent when to trigger it. A quick start section. No fluff.

You can find it in my skills repo.

The important thing about create-skill is the description requirement. The description is the only thing your agent reads when deciding which skill to load. If the description is vague, the skill never fires.

The First Draft

I described what I wanted in plain English.

Something like: interrogate me on what tests to write after a PR, one question at a time, don’t write any code until I’ve confirmed everything.

Claude Code gave me a working SKILL.md in one pass. The structure was solid. Read the diff, identify testable surfaces, ask questions, implement tests.

Then I used it on a real PR and it immediately broke.

What Broke (And How I Fixed It)

It started writing tests after the first question. I’d answer “yes, that’s correct” and it would start implementing. I had to add an explicit rule: agreement during interrogation means “add it to the plan,” not “implement it now.” The interrogation phase and implementation phase are strictly separate.

It asked me things the codebase already answered. I was getting questions about types and return values that it could have looked up. Added a rule: if the codebase answers the question, read the code first.

The temp file path broke in worktrees. I was using /tmp/test-plan.md which is global. When you’re working in git worktrees, plans from different branches bleed into each other. Moved it to ./test-plan.md at the project root so each worktree gets its own plan.

The test plan felt throwaway. A flat temp file gets deleted and forgotten. I moved it to docs/test-plans/001-short-description.md, ADR-style. Now the test plan lives in the repo. Future devs can read it to understand why a test exists, not just what it asserts.

I over-engineered a format template. I created a TEST-PLAN-FORMAT.md with rigid fields (What, Why, Edge Case). Then I realised the agent could just figure out the format on its own. Killed the file.

Every one of these changes came from using the skill on real PRs. Not from upfront design.

The Surprise Feature

When the skill reads the diff, it also inspects the surrounding codebase. If it finds untested code that your changes touch or depend on, it surfaces those gaps too.

I didn’t design this. It emerged from the instruction to “identify every testable surface.” The agent interpreted that broadly and started flagging existing holes. This alone caught two bugs across my 20 PRs that had nothing to do with the new code.

The Final Skill

62 lines of markdown. No scripts. No reference files. Here’s the flow:

1. Run /test-interrogate after your PR is ready

2. It reads the diff and identifies every testable surface

3. One question at a time about intended behaviour

4. Agreed cases documented in docs/test-plans/

5. You confirm the full plan

6. Tests implemented in one pass

7. Suite runs

8. Failures triaged with you: bug in code, bad test, or missing requirement

You can install the skills with this command: npx skills add Alexanderdunlop/skills .

The key constraint is step 4. No test code gets written until every question is asked and you’ve confirmed the plan. This is the rule that makes the whole thing work.

How to Build Your Own

Read the SKILL.md in my repo. It’s 62 lines. That’s the entire skill.

Build your own skill. Start with the create-skill template. Describe what you want in plain English. Get a first draft. Then use it on a real task and fix what breaks.

The pattern is always the same. Use it before you finalize it. The good changes come from hitting real problems, not from planning.