Every product team runs some form of user survey, and a lot of them come back with data that feels informative.But I’d argue the data predicts very little.
So how should you design a survey that helps you make business decisions? In these popular teardowns, I take real surveys from real companies (Superhuman, ChatGPT, Mailchimp, ClassPass) and show what they are doing right and what they are doing wrong.
These teardowns are for you if: you’re fielding a survey or a discovery study soon, or you own research, product, or churn.
Curious about working with Irrational Labs on research design? Reach out: kristen@irrationallabs.com.
Superhuman: Why good beats perfect in survey design
👉 The people who quit your survey are the ones you needed in it
The survey Superhuman sent me was thoughtfully designed, academically rigorous, and completely unusable for anyone with a normal attention span. I couldn’t finish. It asked me to pick my industry from a 50+ option dropdown (nobody scrolls that far; people take the first close-enough answer), then asked how many hours a week I spend on meeting recaps. I don’t track that, so I made a number up.
I had four minutes before my next meeting, and that’s the real context for your survey, not someone with unlimited time. When I dropped off, I didn’t even know whether my partial answers were saved. Every friction point can cost you completions, and the people who bail are your most representative users, which leaves you with superfans. Better to get 500 people through a simple survey than 50 fans through your research masterpiece.
What you’ll learn:
How to avoid self-selection
Recommended and specific question wording for very classic questions
What ChatGPT’s survey reveals about user research design
👉 Throw away those five whys
I found myself in the middle of an accidental behavioral science case study when I took ChatGPT’s survey. It asked whether I could spare “a few minutes,” which I liked, then revealed 5 to 7 after I clicked (it took me nine), and handed me 10+ options on questions I wasn’t reading past the top of, which is how you get skimming and ordering effects.
Then it asked the question researchers love most: why did I upgrade to ChatGPT Plus? Great question. The researcher’s job would be so easy if that was all it took. To answer honestly I’d have to remember what I did weeks ago, diagnose my own motivation, and be willing to tell you about it, so what comes back is a story you can’t distinguish from the truth. Ask what I did yesterday instead, because the recent past is the piece any of us can still report accurately.
What you’ll learn:
Why you shouldn’t ask why questions
A trick to make a longer survey feel a bit shorter
Mailchimp: How not to do customer discovery
👉 When you're surveying users, talk is cheap
On one level, this is a teardown of one feedback survey. It’s also a map for turning traditional research questions into ones that predict behavior, and it covers time expectations, incentive design, and question design. The first two are quick: “this takes 15 minutes” gets you “great, I’ll do it later,” which becomes never, and a $20 reward “for those who qualify,” paid weeks later, motivates less than a small guaranteed one paid now.
Question design is where it gets interesting. “How likely are you to consider a new CRM?” is cheap talk: sure, I’m open to it, and to new HR software too, because agreeing costs me nothing and doesn’t mean I’ll do it. The other question fished for compliments. “How believable is it that this product would free up more of your time?” Extremely believable, thank you for asking! No teeth, so my nice answer says nothing about what I’ll do. Design two versions instead, one that frees up my time and one that makes me money, and force the choice. Otherwise you get persona development, when a dry test would tell you more: put an invoicing promo in the product and see who clicks.
What you’ll learn:
How to squeeze more juice (insights) from your questions with small changes
When to reconsider if a survey is the right thing for your team
How ClassPass gets you to rejoin (or not): Exit survey teardown
👉 The order of your questions decides the answers you get
You own churn, so you send an exit survey, and it opens with a screen of statements about price. That’s what ClassPass sent me after I canceled, and it did what it was built to do: it got me thinking about cost. My actual reason was that there weren’t enough studios near me. I agreed with the price statements anyway, because they were also true, and now that agreement sits in a dataset looking like a reason I left.
It also required an answer on every item, so I filled in things I don’t believe. The primary-reason question, the only one they needed, showed up near the end. Prioritization in survey design is incredibly important.
What you’ll learn:
How to prioritize questions
If/how you could bring back a churned user via the survey
How to do research on yourself
👉 A survey you’ll actually fill out takes 20 seconds
My husband and I send ourselves a survey every week. Our memories are worse than we’d like and our daughter is two and a half, so the whole thing is a gift to our future selves: this was the week she learned to make me coffee, which definitely made the survey. It’s the same diary study you’d run on your users, pointed at ourselves.
You could run one on a new relationship, a new job while you still have fresh eyes, your mood, or the ideas you’d otherwise lose by Thursday. Ours is a Google Form plus a weekly Zapier email subject-lined “Do this now: 20 seconds,” with one rating question and one text box.Critical to keep it short or you won’t do it every week, and doing it every week is the point.
What you’ll learn:
How to set up a weekly self-survey with a Google Form and one Zapier trigger
Six prompts worth tracking, from mood to the ideas you keep losing
That’s it for this best-of edition. Whether you’re fielding a survey this quarter, rebuilding an exit survey, or figuring out how to do research in the world of AI, these five teardowns have some nice behavioral science tidbits that should give you ideas.
Want to work with Irrational Labs on research design? We’ve helped companies like Microsoft, Google, and Intuit design research that predicts behavior. (Microsoft brought us in to build behavioral science into their research team’s protocol, and with Intuit’s payroll team, we ran the experiment that lifted emergency savings 5x). Reach out: kristen@irrationallabs.com.
Have a friend who would enjoy these teardowns? Click the button below to refer them.👇
For more deep dives into product psychology, browse the full archive here. A new teardown drops every week. Subscribe so you don’t miss any.
We design products that change behavior, using behavioral science. Check out our case studies to see it in action.
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.