Continuous Discovery Without the Interviews

Two months ago, in early February, I spent a week reading competitor reviews by hand. A tab open on the App Store, a notes file open on the other screen, a mug of coffee, and a habit of copying out anything that sounded like a frustration. That was early research into who I was actually building for and what they were unhappy about when they bought existing B2B apps for their store.

I’d almost forgotten I did that.

Last week I was building a lead discovery pipeline for the same two apps. The logic is straightforward on paper. A scraper pulls reviews from a list of competitor apps. A scoring step reads each review and flags the ones where the merchant sounds frustrated enough to be worth reaching out to. A matcher resolves the store name to a domain. A later step pulls contact info and drafts an outreach email. I needed leads. This is how you find leads when you can’t afford ads.

Somewhere in the middle of wiring up the scoring step, I remembered February.

The data flowing through the new pipeline was the same data I’d been reading by hand two months earlier. I was building sales infrastructure on top of what had already been product discovery. And Teresa Torres’ continuous discovery model, the one that says good product management requires a steady cadence of customer interviews feeding an opportunity tree you keep pruning, suddenly looked different from this angle. Torres’ framing assumes the customers are yours to interview. Mine weren’t. But they were already talking. Publicly, in writing, with timestamps and star ratings attached. They just weren’t talking to me.

So I set up a second reader on the same pile of reviews.

The lead scoring step asks one question. Is there enough pain in this review to justify contacting the merchant who wrote it? If yes, it tags the review and hands it downstream to the outreach step. If no, the review gets filed as noise. Praise is noise, by definition. A happy customer of a competitor isn’t a prospect.

The second reader asks a different question. Across a list of product themes, what are merchants repeatedly complaining about, asking for, or thanking competitors for? Praise is a first-class citizen here. If fifteen merchants over six months all thank a competitor for flexible wholesale pricing, that’s a roadmap signal, not noise. The second reader sorts reviews into themes, counts how many distinct merchants mention each theme, and scores the result.

Same reviews. Same model underneath. Different prompt. Different output shape. The first output fills an outreach queue. The second output writes to the per-app roadmap files.

That was the plan. It didn’t work.

The first roadmap the classifier gave me was worthless.

It suggested features I’d already shipped. The approval flow for new B2B buyers came back as a top opportunity, weeks after I’d finished building it. It suggested features that had nothing to do with either of my apps. Tax integration for EU merchants. An affiliate program. A loyalty points system. All real things competitors get praised for. None of them things I’m going to build for either B2B Onboard or OrderFlow.

I had thought “we already have the data, discovery is solved.” What I actually had was a prompt that didn’t know what my products were, generating a wishlist that made no distinction between work that was done, work that was in scope, and work that belonged to a different company’s product.

The failure mode is obvious in retrospect. A classifier that sees only the review text can tell you what merchants are feeling. It can’t tell you which of those feelings map to something you can or should act on. The missing input wasn’t more reviews or a better prompt. It was context about the product the roadmap was supposed to be for.

I spent a couple of evenings rewriting the classification prompt. Tightening the theme list. Adding exclusion rules. None of it moved the needle much, because the problem wasn’t how the classifier reasoned about reviews. The problem was that it had no idea what “my product” meant.

The fix was to give it one.

I had Claude Code walk through each app’s codebase and write a short feature profile. What’s shipped, what’s in progress, what’s explicitly out of scope. One file per app, a couple of pages each, written from the code itself rather than from my memory. The classifier now reads the relevant profile before it reads any review. Suggestions get filtered against what the product already is.

The roadmap stopped hallucinating work the same afternoon.

With the profile in place, one of the entries currently sitting on OrderFlow’s roadmap file is “flexible custom pricing per buyer segment.” It came from 154 review signals across 12 distinct merchants, all of them happy with how a competitor handles exactly that. Nobody told me they wanted it. I’ve never interviewed any of those 12 merchants. They were writing to a competitor’s review page about a competitor’s product, and the second reader pulled that pattern out while the first reader was filing the same reviews as praise, not as a prospect.

This is where the piece should end with something tidy about continuous discovery without the interviews. I don’t think I can honestly end it there yet.

The scoring logic has a second half I haven’t been able to test. It gives a confidence boost when the same theme shows up in two independent places, competitor reviews and my own support tickets for the same merchant. Two voices from different directions, about the same gap, is supposed to count for more than one voice twice. Right now that half of the signal is dead, because neither app has launched, which means I have no support tickets, which means every entry on the current roadmap is running on a single source. I’m reading half a validation system and treating it like a whole one.

At the scale I deal with in the day job, customer voice data usually lives in four different tools owned by three different teams, and nobody reads any of it. The solo version of that is that one person reads all of it but is also the person deciding what to build, which is its own failure mode. The hope baked into the two-reader setup is that the second half of the signal arrives with the first merchant who files a ticket, and the roadmap the second reader is writing right now turns out to still look approximately right once it does.

I’ll know in a few weeks.