The Dreaded Case Study

When I was young, bright, and shiny PM, my more experienced superiors would amaze me. I’d bring them the big hairy problems that had stumped me for days, and they would just… unpack them. Zooming out effortlessly, quickly spotting what mattered and what didn’t, showing me how I’d been bogged down in some obscure corner of the problem and had missed the bigger picture entirely.

I wanted to be just like them.

Later I learned the words. First principles thinking, signal vs. noise, pattern matching, frames for decision-making. After many books, and many years of practice, both as a product manager and as a human navigating the world, I think I’ve gotten better at it. Not perfect, but we’re certainly cooking with gas.

So when someone questions the things that sit at the very heart of that, it doesn’t feel great.

That’s why failing three interview processes at the case-study stage really sucked.

Apparently the way I break down problems and arrive at solutions was wrong. I could talk the talk, but I couldn’t walk the walk. And if it happened three times in a row, it had to be me, not them.

I’d played by the rules. I studied the “how to ace product sense interviews” courses, I practiced, I worked my butt off, all to no avail.

Then, eventually, I passed one, and the spell broke.

These days I’m on the other side of the table. At SwitchUp, I’m the one torturing applicants with a case study, and I get tremendous signal out of watching someone approach a problem space similar to ours.

So two things are true: There’s a lot of signal to be gained from case studies, but for applicants, the stake of performing well are so high, that good people fail.

The more I studied how to pass these things, the worse I seemed to do.

If you read the product sense interview guides, you learn a very structured, almost high-school approach. You make your assumptions explicit. You describe the user and the product. You frame the problem, ideate a long list of solutions, score them with something like ICE, and pick a winner.

I’m not knocking the steps, they make a world of sense. When we don’t take a structured approach, but skip to the end, we tend to end up with “the first idea that comes to mind”. There is a TON of research explaining why that first idea is rarely the best idea. Here, here, and here.

However, following those rules so neatly forced me into a weird straightjacket, leading me to a solution that I could feel wasn’t *it*. The formal “product sense” path led me in a completely different direction than my instinct would. And the hiring committee could sense it.

So which path is right?

This actually goes into a larger question: structured approach vs. instinct.

We went from “gut feel is for idiots, data is the new oil” (anno 2015) to “Not data-driven, but data-informed” (anno 2020) to “Product sense, taste, conviction are king” (today).
We went from OSTs and GIST boards (I’m still a big fan of both!) to “principles over frameworks”. So are these things truly opposites that can’t peacefully co-exist?

I don’t think so.

My instinct is not random guesswork. It’s a mental model of what a good product looks like, *built by following the steps*.

After years and years of interviewing, observing, framing problem spaces and testing solutions, I’ve internalized some fundamental product-builder truths:

• Look for market pull, rather than try to educate the market. Innovation for innovation-sake hardly ever works.

• Build for your target audience, and no one else.

• If there’s an important knowledge gap, fill precisely that gap with the strongest evidence you can get *fast enough*

• Work backwards from the result you wan

• If you want someone to buy your solution, look at what they do today and make it undeniably better, more fun, or less shitty.

• For the love of god: Don’t overcomplicate stuff

Especially when I’m building for an audience I know quite well, my spidy senses tell me what will work. It’s not guesswork, it’s instinct.

Conviction is not a shortcut around the work. It is the accumulated residue of having done the work, many times over.

Every committee comes to the table with their own frame of what they’re trying to learn, and they’re not always explicit about it.

Some case study rounds really are just looking at clarity of reasoning.

Some really do want you to follow those rigid product-sense steps exactly.

Some of them secretly hope you’ll arrive at the specific solution they already have in mind. (That’s is silly thing to do, and if you fail it, it really isn’t on you).

The committee where I finally passed didn’t care about rigid processes or frameworks at all. They wanted to see if I had the pattern-matching and confidence to back my own conviction without jumping blindly to the very first idea that comes to mind.

So where did that leave me in practice? I’m still working this out, and I’m not sure it qualifies as wisdom, but here’s what I did to pass that case:

I ran two paths and compared them.

The first path followed the product-sense process strictly: assumptions, audience, a problem to solve, a wide set of solution ideas, scoring, and a single winning solution.

The second path started with my instinct, let it set the general shape of where the answer should go, and then used the same process to generate and compare variations around that shape.

I recently watched Mark Pincus on Lenny’s podcast, and the thing that really stuck with me was “Your instinct is right 95% of the time but your idea is right at best 25% of the time”. I let conviction dictate the high-level shape of the thing, but then I went broad on ideating different variations on it. This second path also led to a single winner.

I compared the winners of the two paths, and asked myself “ which one actually fulfils the assignment best?”. In this case the instinct-led path won, so I doubled down on it.

• Failing a case study (multiple times over) does not mean that you’re bad at thinking. Unfortunately, many hiring committees aren’t trained at hiring, and come in with a preconceived notions on exactly how thinking should work for everyone, and what the answer to the question should be. In better cases, you’re not only being judged for your process, but also on other factors (e.g. domain expertise). And yep, sometimes, your thought process was indeed a bit incoherent. It happens.

• The product sense straightjacket is not the right tool for everyone, for every interview. Most hiring committees want to see who you really are, how you really work. So trust yourself.

• Instinct and process aren’t at odds. Instinct follows a strong mental model, and following process builds it over time.

• Use instinct to find the general direction, use process to ideate multiple variations. In the end, pick the thing that solves the assignment best (or “addresses the #1 opportunity best” or “likely has the highest impact on the #1 metrics you’re trying to drive”).