Your Test Suite Is Lying to You

Last week left a condition hanging: that whole refactoring discipline only works if the tests are worth running. That isn’t a safe assumption, and it fails for a reason that has nothing to do with how many tests there are.

The build is green. Coverage is 83%, comfortably above the gate the team agreed on two years ago. And the last three production incidents all happened in lines the report counts as covered.

Nothing in that report is false. Every one of those lines really did run while the tests ran. The lie is in the sentence we say out loud when we read the number: “that code is tested”. Coverage never promised that. It promised execution, and we heard verification.

This post is about the gap between those two words, and about the measurement that shows how wide ours already is.

A coverage tool instruments the code and records which lines and branches ran while the suite executed. That’s all it does. It doesn’t read our assertions, and it doesn’t try to, because an assertion is a method call like any other and nothing marks one that checks a result apart from one that checks nothing.

So the number answers a narrow question. Not “is this behavior verified” but “did this line run at least once while the suite was running”1. Only one of those is about whether our tests would catch a bug.

Here’s a rule small enough to fit in a post and real enough to have bitten me. A library loan can be renewed, with three refusals: not if it’s already overdue, not more than three times, and not if another member has reserved the title. A renewal moves the due date fourteen days out.

And here's the test that ships alongside it more often than anyone likes to admit:

That test asserts nothing. It calls the method and ignores the return. It passes if renew extends the loan by fourteen days, and it passes if it extends by fourteen hundred2. It passes if the body is deleted down to return this. The only thing it proves is that the call doesn’t throw3.

I’ve seen this test written by an IDE template, by a code generator, and by a tired human on a Friday evening. I’ve also seen it written by an AI assistant asked for “a test for this method”, which produced exactly that, with a green check next to it. It got my approval a comment from me two days later, once I read it properly.

Coverage does catch this one, and it’s worth being exact about how. Of the seven executable lines in renew, four run: the three conditions and the return statement. The three throw lines never execute, so the report comes back at 57% line coverage and 50% branch coverage for the method. That’s a low number, and telling us about a low number is the one thing coverage is genuinely good at.

So the team does the obvious thing and raises it.

The suite below closes every branch. One assertion added to the happy path, one test per refusal.

renew is now at 100% line coverage and 100% branch coverage. Four tests, all green, every branch taken both ways. By every number the build reports, this method is finished.

At least two of its rules are still unverified.

The first is the limit itself. Our limit test uses a loan with five renewals, well clear of the boundary, because five was easier to type than three and nothing in the suite cared. Change renewalCount >= MAX_RENEWALS to renewalCount > MAX_RENEWALS, quietly allowing a fourth renewal, and all four tests stay green.

The second is the count. No test reads renewalCount after a renewal, so the code could decrement it instead of incrementing it, and nobody would find out until a member renewed the same book eleven times.

Closing those two gaps takes more tests, but none of them increase coverage since it’s already 100%. Coverage can’t reward the work that matters most, because it maxed out before that work started. Meanwhile, a developer who adds a getter and a test that calls it moves the project number up.

Renewal doesn’t stop at the aggregate. There’s a service around it that loads the loan, saves it, and writes an audit entry. Usually we write similar tests for that service:

There’s an assertion. There’s a real scenario. And it still says almost nothing about renewal, because what it checks is that one collaborator was called with one string. Change fourteen days to forty and it stays green. Rename the event to LOAN_RENEWED and it goes red, even though nothing a library member could observe has changed.

A test earns its place by failing when the behavior changes, and only then. This one is blind to the change that matters and loud about the change that doesn’t. The second half of that’s a design problem, and out of scope for this post.

That difference is invisible to the coverage number, because a test that asserted the new due date would execute exactly the same lines, and lines running is all the instrumenter sees. It’s worth having two words for the gap. Test coverage is what the tool reports: which lines ran. Behavior coverage is which rules of the domain we’d notice breaking. Nothing in a default build reports the second one, which is why it drifts.

There’s a way to measure the second one, and the idea fits in a sentence. Break the code on purpose, then see whether the suite complains.

A mutation testing tool takes the compiled code and makes small, deliberate changes to it:

• changes >= to >

• negates a condition

• replaces a return value with null

• removes a call to a void method

Each altered version is a so-called mutant, and for each one the tool runs the tests that touch that code. If at least one test fails, the mutant is killed, which is the good outcome: the suite noticed. If every test still passes, the mutant survived, which means our test suite can’t detect that behavior change.

On the JVM the mainstream option is PIT4. Point it at Loan with the four green tests in place, read the renew part of the report, and it sorts into two halves.

Killed:

• Each of the three conditions becomes its negation, and the happy-path loan isn’t overdue, isn’t at the limit, and isn’t reserved, so every one of them makes renew throw and the test fails.

• The return value becomes null, and the happy-path test reads the due date off the loan it gets back.

Survived:

• renewalCount >= MAX_RENEWALS becomes renewalCount > MAX_RENEWALS, and no test uses a loan at exactly three renewals, so nothing notices that a fourth is now allowed.

• renewalCount + 1 becomes renewalCount - 1, and no test reads the count5.

PIT finds six mutants in renew, and the four tests kill four of them, on a method with 100% coverage6. The two survivors are the two rules we worked out by hand a few paragraphs ago. The difference is that we spent ten minutes on one method, and the tool does it for every method in the package while we get a coffee.

Two honest costs and two limits, because adopting a tool without them gets abandoned in week three.

It’s slow. Roughly speaking, the tests run once per mutant, so a class with forty mutation points means forty runs of the tests that reach it. On a large module that turns a two-minute build into something nobody will wait for7. The answer is scope. Run it on the packages that hold the rules rather than on the whole repository, and schedule it nightly or on demand.

It also produces mutants that can’t be killed. When a change produces a program that behaves identically, no test can tell the two apart, and the tool reports a survivor that isn’t a gap. These are called equivalent mutants, and they’re a genuinely undecidable problem rather than a defect in the tool8. In my experience they’re a small part of the survivor list while a suite still has real gaps. As the gaps close, they become most of what’s left.

There’s a second group of survivors we should leave alive: mutants in code we deliberately don’t test. Delete a log statement and nothing goes red, and that’s the behavior we want, because a suite that pins log lines fails every time somebody rewords one. Same for a toString written for a debugger, or a metrics counter nobody makes decisions on. The report can’t tell those apart from a real gap, since it only knows that nothing objected. We answer them by narrowing what the run mutates, not by writing tests to kill them9.

The first limit is on what it can see at all. It only asks the questions its mutators know how to ask, and the mutators are blunt on purpose. Nothing in our four tests pins down whether a loan due today can be renewed today. The default mutators skip it too, because they work on numeric comparisons and isAfter hands back a boolean. That third gap stays invisible to both numbers.

The second limit is over-specification. Mutation testing measures blindness, so the decrementing mutant shows up as a survivor that nothing in the suite objects to, the verify test included. The same test going red on a rename nobody could observe produces no survivor at all. A developer chasing kills is one verify away from making that half worse.

None of that makes coverage useless. Martin Fowler made the case years ago and it still holds. Coverage is a simple way to find code nobody tested, and loses it the moment the percentage is read as a verdict on the tests themselves10. A file at zero percent is real information. A package that falls from 70 to 40 in a single pull request is real information.

Low coverage proves there’s a problem. High coverage proves nothing. Use coverage as a tool that catches untested files. If we want a number in the build config that says something about the tests, the next section is where to get one.

This is worth doing, and one afternoon covers it.

1. Pick the class with the most business rules. One class, and preferably the one where a wrong answer shows up in a user complaint instead of a stack trace.

2. Run PIT scoped to it, then read the survivors. Both plugins filter on classes and on tests, so point them at the class and its own tests and the run takes minutes. Work through the list, one survivor at a time. Each one is a statement: this rule could change and no test would notice.

3. Write one test per survivor that matters, and keep both numbers. Skip the equivalent mutants, and the ones in code that genuinely don’t matter. Then put the class’s mutation score next to its line coverage.

The score has exactly one job here, which is why the third step keeps it: standing next to the coverage number for the same class. That pair is what turns a personal fix into an argument. One class, two measurements, and the gap between them says more than any amount of explaining.

Coverage tells us a line ran, never that it was checked. What we meant is behavior coverage: which of our rules the suite would catch breaking. Mutation testing measures it by breaking the code on purpose and asking whether anybody complained. Run it on the class that holds the rules, read the survivors, and a green build starts meaning something again.

If the team’s testing strategy is a coverage threshold in the build config, send the following to the person who set it:

This article explains why a coverage percentage says less about test quality than it looks. Coverage counts how many lines of code ran while the tests ran. Whether those tests verify anything is a separate question it doesn’t answer, so a suite can hit 100% and still have gaps.

What matters is behavior coverage: which of our business rules the suite would catch breaking. Mutation testing measures that, by changing the code on purpose and reporting how many changes our tests missed. It takes an afternoon on one class, and if that score is far below our coverage number, we know why production incidents keep happening despite the green builds.

A green build should mean the suite looked at this and had no objection. For a lot of suites it means only that the code ran with the suite watching. One afternoon is enough to find out which one we have, and last week’s tidying depends on the first kind.

Media attributions:

• Cover image by the author (generated with Gemini)