Inconclusive...?

This is Part 2 of this chapter. If you missed part 1, read it first! You can find it right here.

I want to tell you about a will.

Several years ago I was running marketing on a digital product that helped people create their own wills. We built a campaign around the messaging “quick and easy.” Our preliminary research told us that adults who needed wills cited “convenience” as an important factor in their decision. So we leaned into convenience. Quick. Easy. Done in fifteen minutes. Off your to-do list forever.

We ran the experiment. Tested it properly. Held control groups. Watched the numbers.

The results came back…inconclusive.

I want to pause on the word inconclusive for a minute, because if you’ve ever run an experiment, you’ve been here. The number didn’t move. The test didn’t fail and didn’t win. The campaign lift was statistically indistinguishable from the baseline. By every standard interpretation, the experiment was a wash.

I’ve taught experimentation to thousands of people over the years, and the most common mistake I see is treating "inconclusive" as "no result." Inconclusive, when you’ve taken a real swing, meaning not just changed a button color or something similarly benign, almost always means something moved. Likely, two things moved in opposite directions and canceled each other out in the aggregate. The signal is in there. Somewhere.

Knowing this, I pulled the thread.

We re-cut the data by complexity of situation. And what we found was that the campaign had not been inconclusive at all. It had been successful with one segment and actively repellent to another. The segments had canceled each other out in the headline number.

Under single, no kids, simple asset situation: the messaging crushed it. They loved quick and easy. Conversion rates jumped. (Age really didn’t matter.)

Married, mortgage, investments, and all that jazz: the messaging failed. Not just failed; failed spectacularly. Which is always helpful, because it’s a signal of what to look for: where there’s smoke, there’s fire. So we did the thing companies should do considerably more often. We got on the phone with some of the customers who had bounced, and we asked them why.

What they told us was obvious, but only in hindsight.

Their objection wasn’t on the surface. None of them could articulate it cleanly at first. But when we stayed in the conversation long enough, the answer kept showing up in different words: Quick and easy made me think you don’t understand my situation. My estate isn’t quick and easy. I have kids and a house and a business. If it can be quick and easy for me, you don’t know me.

(Imagine taking your complicated taxes to your accountant, and five minutes later he emails you: "All done!" Do you trust that accountant? Of course you don't.)

Share

The same campaign, with identical product, price, features, and landing page, was producing two opposite reactions in two very different customers who looked nearly identical on a demographic spreadsheet.

The simple buyer was hiring our product to check a box. The job she was buying us for was: stop feeling like an irresponsible adult who has put this off too long. Quick and easy was the perfect promise.

The complicated buyer was hiring our product to protect everything she had built. The job she was buying us for was: prove to me that you understand the weight of what I am trying to protect. Quick and easy signaled the exact opposite. It signaled superficiality. It signaled that we hadn’t thought about her situation. It signaled, accurately, that we were trying to optimize for the wrong customer.

Two completely different jobs. Two completely different felt needs. Both sitting inside what our demographic persona had cheerfully called “adults who need a will.”

* * *

This is what the original “persona” looked like:

Adults 25-65, household income $50k+, some assets or dependents, interested in financial planning, values convenience.

This is what one of the real personas looked like after we rebuilt them:

Has been putting this off for three years. Knows he should have done it after his first kid was born and didn’t. Most recently thought about it when a friend died unexpectedly at 47 and the estate took eighteen months to settle. Wakes up at 3 a.m. occasionally and thinks about what would happen to his family if he didn’t make it home from a business trip. Has not had a serious conversation with his wife about this. Is afraid the process is going to be complicated, expensive, and require him to make decisions he doesn’t feel qualified to make. Is also afraid that if he uses the cheap easy version online, he’ll do it wrong and discover the mistake too late. Hires the product not to save time, but to feel certain that the thing he loves most is protected.

Your copywriters will LOVE you for the second one.

You can write headlines from it. You can record videos from it. You can train salespeople with it. You can design a landing page around it. You can choose channels around it. You know what he is afraid of, what he is avoiding, what he wishes were true, and what would make him finally pull the trigger.

You cannot use the first one for anything except a slide. Which is, in fairness, the only place it ever actually gets used. (No shame. I’ve done it too. Many times.)

The persona that lives in your shared drive is almost certainly the first one. The persona you actually need is the second one. The gap between those two documents is where the deals you should have won keep going to a competitor who, by every objective measure, has a worse product than yours.

* * *

I have made that sound almost easy. Pull the data, make a few calls, rebuild the persona, win.

It was, emphatically, not easy. That re-cut of the Will numbers, the phone calls, the rebuilt persona, none of it happened by accident.

I’ve spent a large part of my career doing exactly this kind of work. Sitting in on user interviews. Running attitudinal surveys. Reading open-ended survey responses until my eyes crossed. Watching screen recordings of real people using a product and narrating their confusion out loud. If you asked me how many times I have personally watched a customer do something that contradicted everything the company believed about them, the honest answer is that I stopped counting somewhere in the hundreds.

What all that volume teaches you is that the Will result was not unusual. It was the rule. The surface answer is always wrong. The felt need is always buried. The interview always surprises the people who were most certain they already knew their customer. Remember what we learned in the last chapter: your customers are lying through their teeth without even knowing it.

It also teaches you that there are two kinds of customer research, and most companies are only doing one of them.

***

Quantitative research (some surveys, analytics, conversion data, the re-cut of the Will numbers) tells you what is happening. It is broad, it is statistically reliable, and it is completely silent on the question of why. The numbers told us the complicated segment bounced (and even getting to “complicated situation” took effort). The numbers could not tell us, and never would have told us, that the reason was a single word, “easy,” landing as an insult.

Qualitative research (interviews, conversations, watching actual humans struggle with your actual product) tells you why. It is narrow, it does not scale, you often cannot put a confidence interval on it, and it is the only thing on earth that will ever surface a felt need. “My estate isn’t quick and easy” did not come from a dashboard. It came from a person, on a phone, given enough room to talk past their first three answers.

You need both, and you need them in the right order. The quantitative tells you where to point the qualitative. The qualitative tells you what the quantitative actually meant. Run only one, or run them out of sequence, and you get exactly what we got the first time: a result that says “inconclusive” and a team that shrugs and moves on.

Here is the part that should concern you. You are almost certainly being handed research right now that is quietly garbage, and you have no way to tell.

I’m not going to teach you to run interviews (at least not in this book…sorry, we only have so many pages). That is an art and it takes years to get good at it. But I will teach you how to tell, in about ninety seconds, whether the research your team just set in front of you is worth the paper it is printed on. There are a few tells.

The first is the leading question. “How much did you enjoy our new onboarding experience?” is not a question. It is a request for a compliment. It has told the respondent what answer you want before they have opened their mouth. Real research asks, “Walk me through the last time you set up a new tool. What happened?” and then waits. If you read your company’s last customer survey and the questions sound like they already know the answer, there you go.

The second is the double-barreled question. “Was our product easy to use and reasonably priced?” That is two questions wearing one trench coat. The customer who found it easy but overpriced has no way to answer honestly, so they pick one, and you will never know which. Every double-barreled question on a survey produces a data point that means nothing and gets reported as though it means something.

The third one is the most dangerous, because it feels so reasonable: asking the customer to predict their own future behavior. “Would you buy this?” “How likely are you to recommend us?” We spent the entire last chapter establishing that people cannot reliably tell you what they will do. They tell you what they wish they would do. A survey built on self-forecasting is collecting aspirations and labeling them predictions. The last chapter gave several examples of how this can fall apart.

The fourth is not a flaw in the questions at all. It is a flaw in the room. The single most common way a company talks itself out of the truth is the interviewer who cannot tolerate silence. The real answer is almost never the first thing out of a customer’s mouth. The first answer is the polite answer, the rational-sounding answer, the press release from the last chapter. The true answer arrives third, after a pause the interviewer has to be disciplined enough to allow.

Most people, desperate to fill the dead air, leap in with a follow-up and lead the witness straight past the thing they came to find. Remember: the ellipses-heavy phrasing in the drill-bit section a few pages ago was not a stylistic tic. That is roughly what a good interview actually sounds like. Long, uncomfortable, full of pauses, the real answer surfacing somewhere around the fourth “well…”; which is, for my money, the single best moment in all of market research. A stranger deciding, in real time, to trust you with the true thing.

None of these are exotic. They are the everyday texture of bad research. And bad research is worse than no research, because no research at least makes you cautious. Bad research makes you confident.

And armed with a pretty slide deck, you will confidently drive your business off a cliff.