You can’t A/B test your way to PMF

There is a quote by the founder of Notion about building new products that I’ve found myself coming back to:

“Too much of yourself, then there’s no users. You’re just doing an art project.
And too much business, you’re building a commodity.”

I think this captures one of the most important tensions in product development. And today, I want to argue that in the early days of any product, it is usually correct to stay closer to the art project than the commodity side of that spectrum.

Intuition tends to pull you toward art. Data tends to pull you toward commodity.

Product–market fit (PMF) lives somewhere in between.

In my experience, the teams that win are willing to sit on the art side longer than it feels safe.

Before you reach PMF, the product is a little bit of a mess. It changes frequently. It’s used in ways that aren’t repeatable. You yourself do not have clarity how it will look like in a few months, in turn, your product road map is proportionate to how long the product has been in existence. Even if you manage to deliver value early, and even if you find a way to monetize, it’s often difficult to turn that into something truly productized. This is why so many people end up running consulting businesses or agencies. Turning something useful into a scalable product is hard.

This reality has an important consequence: early data is sparse, noisy, and misleading. It lags reality.

I’ve never heard of a team A/B testing their way into product–market fit.

So if data can’t reliably lead you there, what do you do instead?

You rely on intuition, or what people now often call taste.

Intuition sets direction. Data becomes one form of feedback, but not the only one. Your understanding of how users actually behave, where they get confused, the questions they ask, how the industry works, how your solution fits into existing workflows, these are all signals, alongside quantitative metrics, they inform and refine your intuition over time.

Despite this, many people believe that without sufficient data, you shouldn’t make a decision at all.

In practice, data often becomes a way to outsource accountability. This shows up most clearly in two places. The first is among people early in their careers, who haven’t yet built the judgment needed to operate in messy, uncertain environments. The second is in large organizations, where accountability is diffused, and decisions are driven by consensus rather than ownership.

In both cases, data provides cover. If a bet works, great. If it fails, “the data misled us.” From that perspective, asking for more data is rational self-protection.

Another problem with over-relying on data is that the 0 to 1 phase is often pre-data. The most important questions at this stage are qualitative:

• Is this a real pain or just a nice-to-have?

• Is the time to value short?

• Are users disappointed if the product is taken away?

• What do they replace it with today?

Dashboards don’t answer these questions, not until you’ve shipped enough, learned enough, and orchestrated the product correctly. And deciding what to instrument in the first place requires judgment.

There’s a simple way to test the idea that data alone leads to good decisions: if that were true, large companies would launch successful products consistently. They have enormous amounts of data. And yet even the best-resourced organizations, think of Google, Apple, OpenAI, etc., have many misses.

The reason is that large organizations tend to optimize for value preservation rather than value creation. Metrics become targets (Goodhart’s law). Having data is not the same as having clarity.

Startups, almost by definition, are games of outliers. You win by making non-consensus bets that turn out to be right. If your advantage is that you see something others don’t, early data will not give you strong confirmation. What you’ll see instead are weak signals and skepticism.

In team discussions, I often hear some version of: “We should be more data-driven.” My response is usually that data is extremely useful if three things are true:
- first, you actually have enough of it;
- second, you trust that it’s set up correctly;
- and third, it aligns with your gut feeling, informed by talking to users and your collective experience.

If any of those are missing, I prefer to rely on intuition. Not because intuition is magic, but because it is informed by hundreds of conversations, many failed experiments, and prior experience. This matters more in the early days.

In the 0 to 1 phase, you don’t use data to decide. You use data to learn whether your thesis is wrong.

There are, of course, times when it makes sense to move along the spectrum from art toward commodity.

If your intuition is often wrong, you need tighter feedback loops and the discipline of “strong opinions, loosely held.”

Once you have a working loop, activation, habit, retention data becomes extremely powerful. It should lead decisions around onboarding improvements, growth experiments, performance, and pricing. At this stage, taste and data complement each other well. But not before the loop exists.

There are also cases where the organization becomes large enough that intuition no longer scales easily. Even then, the underlying problem is often cultural rather than a simple choice between taste and data.

In most other situations, following your gut leads to better outcomes. Decisions create motion. Motion generates information. Information, both qualitative and quantitative, feeds back into judgment and helps you navigate the idea maze.

If you wait for data to justify every decision, you stall. And without momentum, you end up with very little data anyway.

So my advice to people early in their product careers is simple: if you don’t yet have strong intuition, defined as being more right than wrong over time, work with someone who does. Study how great products are built. Pay attention to why decisions are made, not just which metrics are cited.

Eventually, you’ll develop the judgment needed to push through the 0 to 1 phase. At that point, data can take on a larger role. And only then can you build a culture that genuinely balances taste and data, rather than hiding behind one or the other.