Make Your Bug Report Shrink Itself: Test-Case Reducers

A bug only reproduces on a giant input. A 5,000-line file. A program your fuzzer generated. A 200-line config where something is wrong. You know the bug is in there; you just don’t know where.

So you do what everyone does: open the input, delete a chunk, re-run, see if it still breaks, undo if it doesn’t, try a different chunk. An hour later you’re squinting at a slightly smaller mess, and you’ve quietly accepted that deleting region A doesn’t help but deleting A and B does — and now you’re contemplating the combinatorial nightmare of which disjoint pieces to remove together.

There’s a tool for this, and most developers have never used it. It’s called a test-case reducer, and it will take that 5,000-line input and hand you back the five lines that actually matter — automatically, and without understanding your bug in the slightest. (The framing here follows Laurence Tratt’s writeup, which is the best argument I’ve seen that these tools are badly underused.)

The idea: keep deleting, keep checking

A test-case reducer takes three things: your program, the failing input, and one small thing you write called an interestingness test. Then it does something almost insultingly simple — it tries shorter and shorter versions of the input, and for each one it asks the interestingness test a single question: does this still trigger the problem I care about? If yes, it keeps the smaller version and keeps going. If no, it backs up and tries a different cut.

That’s the whole loop. And it works astonishingly well: reductions of 95–99% are routine. The input that took you an hour to even start reading becomes something you can take in at a glance.

The one piece you write: the interestingness test

The reducer does the shrinking; you do exactly one thing — tell it how to recognize the bug. By convention, the interestingness test is a program (a shell script is fine) that exits 0 if the input still reproduces the bug, and non-zero if it doesn’t. That’s it.

Say your program segfaults on a bad input. Your interestingness test runs it and checks for the crash:

#!/bin/sh
# exit 0  -> still interesting (still crashes the way we care about)
# exit 1  -> not interesting (don't keep this reduction)
./myprogram "$1" 2>&1 | grep -q "Segmentation fault" && exit 0
exit 1

This little script is where all of your knowledge about the bug lives. “The bug” is whatever this test says it is — a specific crash, a specific wrong output, a specific log line, an assertion that fires. The reducer never needs to know any of that. It only needs a yes/no.

Why it works: the reducer is gloriously stupid

Here’s the part that trips people up, and it’s worth sitting with, because it’s the whole trick. The reducer has no idea what your program does. It doesn’t parse your input, doesn’t understand your language, doesn’t know why any particular line matters. It deletes something, runs your test, and reads one bit back: still interesting, or not.

That ignorance is not a limitation — it’s the source of the power. Because the reducer understands nothing, it works on anything: C source, a JSON document, a SQL query, a sequence of recorded API calls, a binary blob. You never teach a reducer about your bug. You teach it, via the interestingness test, how to recognize your bug, and then its total cluelessness about everything else lets it grind through a search space no human would have the patience for. “Does this smaller thing still break?” asked ten thousand times is something a computer is very good at and a person is very bad at.

Build one in a dozen lines

The fastest way to stop being suspicious of reducers is to write one. Here’s a complete (if naive) line-deleting reducer in Python. It takes the interestingness test as its first argument and the input file as its second:

#!/usr/bin/env python3
import subprocess, sys, tempfile

test, path = sys.argv[1], sys.argv[2]
lines = open(path).read().splitlines()

i = 0
while i < len(lines):
    candidate = lines[:i] + lines[i+1:]          # the input without line i
    with tempfile.NamedTemporaryFile("w", suffix=".txt") as f:
        f.write("\n".join(candidate))
        f.flush()
        still_interesting = subprocess.run([test, f.name]).returncode == 0
    if still_interesting:
        lines = candidate                          # keep the smaller version
    else:
        i += 1                                     # that line was needed; move on

print("\n".join(lines))

Walk it through: it loads the input as a list of lines, then repeatedly tries removing one line at a time. If the test still passes (exit 0) without that line, the line was inessential, so it’s gone for good. If the test fails without it, the line mattered, so we keep it and move to the next. When it has tried every line, what’s left is an input where you can’t remove a single line without losing the bug. That’s a real test-case reducer, and you can run it on any text input you have.

It’s slow — one program run per line per pass — and it only deletes whole lines. But it works, and seeing it work is the point: there is no magic in here, just “try removing things, keep what still breaks.”

The fixpoint trick: don’t stop too early

That first version has a weakness. Once it passes line i, it never reconsiders the lines before it — but removing a later line can unlock an earlier one. Maybe line 3 couldn’t be deleted while line 40 was present, but with 40 gone, 3 goes too. A single pass misses those.

The fix is one idea: after any successful deletion, start over from the top. Keep doing full passes until an entire pass removes nothing — a fixpoint. In our reducer that’s a one-line change, resetting the index whenever we shrink:

    if still_interesting:
        lines = candidate
        i = 0            # something changed — re-examine everything
    else:
        i += 1

Now it keeps chewing until it genuinely cannot remove anything more. It’s slower, but it reduces much further, and “much further” is exactly what you want when the goal is the smallest possible input.

In practice, use a real one

Writing your own is the way to understand reducers; it is not the way to use them. The naive line-deleter above is thousands of times less effective than the real tools, which do far cleverer things: they remove whole functions and blocks, simplify expressions, rename things away, balance brackets, and — crucially — run many candidates in parallel.

A few worth knowing:

• C-Reduce (creduce) is the canonical one, built by and for compiler developers, and it’s brutally effective on C and C++. The community that hits the gnarliest reduction problems — “this 200,000-line preprocessed file miscompiles” — lives here.
• shrinkray is a modern, language-agnostic reducer that works well on arbitrary text and structured formats, not just C.
• The underlying algorithm has a name — delta debugging (ddmin) — and classic delta tools implement it. It’s the formalization of “try removing chunks, halving the granularity when you get stuck.”

With any of them, you still write the same thing you wrote above: an interestingness test that exits 0 when the bug is present. You bring the recognizer; they bring a vastly better search.

The deeper trick: reduce toward anything, not just “smaller”

Here’s the idea that turns reducers from a neat trick into something you’ll reach for constantly, and it’s the part Tratt’s piece builds toward. By default a reducer minimizes length. But “interesting” is whatever your test says it is — so you can make the test reward other properties, and steer the reducer toward them.

Suppose two builds of your program disagree — FAST=0 prints one thing, FAST=1 prints another, and only one is correct. An interestingness test for that doesn’t just check for a crash; it compiles both ways and asserts the outputs differ:

#!/bin/sh
set -eu
cc -DFAST=0 -O2 "$1" -o slow
cc -DFAST=1 -O2 "$1" -o fast
test "$(./slow)" != "$(./fast)" || exit 1   # interesting only if they disagree
exit 0

The reducer now shrinks toward “the smallest program where the two builds still disagree” — a far more useful target than just “small.” You can push this further: make the test also require that the program runs fewer than N instructions, or that an error happens at least K times, or that a value stays within some range. Every extra condition you put in the test is a knob that bends the reducer toward the most informative minimal case, not merely any minimal case. The reducer is still gloriously stupid; you’ve just given it a sharper definition of “interesting.”

The catch: the reducer will cheat

A reducer is a tireless optimizer with no judgment, and that combination means it will happily exploit any loophole in your interestingness test. This is the one thing that bites people, so let’s say it plainly: a reducer reduces toward whatever your test accepts, which is not always the bug you meant.

The classic trap is a test that’s too loose. If your test just greps the output for error, the reducer may shrink the input down to something that produces a completely different error that also happens to contain that word — and now you’ve lovingly minimized the wrong bug. The fix is to make the test as specific as you can stand: match the exact assertion message, the exact exit code, the specific crash address — not a vague substring.

Two more guards are worth building in. First, handle inputs that don’t even run: a reduced program might fail to compile or hang forever, so wrap the run in a timeout and treat a non-compile or a timeout as uninteresting (exit non-zero). Second, in C and C++ especially, watch for undefined behavior — a reducer will cheerfully reduce a program into one that “crashes” only because it now reads uninitialized memory, which isn’t your bug at all. The standard defense is to have the interestingness test also run a sanitizer (or a UB checker) and reject anything that trips it. A few extra || exit 1 lines in the test save you from a confident, minimal, wrong answer.

Make the test fast — it’s the inner loop

One practical thing decides whether a reduction takes ten seconds or all night: the interestingness test runs thousands of times, so its speed is the whole reduction’s speed. If a single check takes five seconds, ten thousand candidates is most of a day; get it to half a second and the same reduction finishes over lunch.

So treat the test like the hot path it is. Strip it to the minimum that’s still reliable — skip slow setup, don’t spin up a container if a direct call works, keep a tight timeout so one hung candidate can’t stall the run. And lean on parallelism: the real reducers evaluate many candidates at once across your cores, which is one more reason to graduate from the single-threaded toy above to creduce or shrinkray once the input is big. The reducer’s job is to ask your question ten thousand times; your job is to make the question cheap to answer.

Where this earns its keep

Reach for a reducer whenever a bug reproduces on something too big to read:

• Fuzzer output. Fuzzers are brilliant at finding crashes and terrible at explaining them — they hand you a 4 KB pile of random bytes. A reducer turns it into the three bytes that matter.
• Compiler and interpreter bugs. The home turf: a huge program miscompiles or mis-runs, and you need the minimal snippet to file a bug or find the fault.
• Flaky or slow tests. Reduce a giant failing test input to the core that still fails; combine with an “executes few instructions” condition to also make it fast.
• “Which line breaks it.” A 200-line config, a 5,000-row CSV, a long migration — let the reducer find the rows or lines that actually trigger the problem.

The common thread is that you have a big input and a clear yes/no notion of “still broken.” Whenever both are true, manual deletion is a waste of your afternoon.

Teach it to recognize the bug; let it do the rest

The reason test-case reducers feel like cheating is that they invert the work. Normally you do the searching — reading, deleting, re-running, reasoning about which parts matter. A reducer flips that: you write down, once, how to recognize the bug, in a script that returns one bit, and then the tireless, clueless search is free. You don’t have to be a compiler wizard, and you don’t have to understand why any particular line matters — that’s precisely the understanding the reducer does without.

So the next time a bug only shows up on something too big to read, don’t start deleting by hand. Spend five minutes writing the interestingness test — “exit 0 when it’s still broken” — point a reducer at it, and come back to an input small enough to actually understand. The machine is happy to ask “does it still break?” ten thousand times. Let it.