I was contemplating switching from paperbacks to Kindle even though I love paperbacks and I’ve been resisting for some time. My best friend owns a Kindle and loves it, so naturally I told her I was interested and asked her to tell me why she’s so happy with hers.
She gave me a list: you can always have many books with you without dealing with the storage problem, you can read while taking a bath without worrying about ruining your book, it’s easy to hold, and so on. I thought, oh, she’s really using this Kindle for a lot of things.
Fast forward a few days. She tells me she started a new book that I’d been persuading her to read for a while, and she’s very curious about it. Naturally, the next day I ask: “How is the book?”
She says she took a long bath so didn’t read that much.
I’m shocked. “How come? Didn’t you say you read in your bathtub because you have a Kindle?”
She laughs. “No, I said you could read in the bathtub. Sounds like something you would do, but I don’t read in the bathtub.”
I’ve been interviewing users and influencing business strategy based on my question design and analysis skills for about 10 years, and I’m still absolutely amazed by her answer.
Without my researcher hat on, I made a classic mistake: I took an answer and assumed what it meant for her based on what it meant for me. I heard “you can read in the bathtub” and translated it to “she reads in the bathtub because she has a Kindle.”
This happens all the time when research planning is rushed or not done properly. And here’s the thing that bothers me about most advice on research: we focus obsessively on asking better questions while barely talking about interpreting the answers we get.
The Question Design Obsession
I see frameworks everywhere on how to ask better questions. They’re useful for practicing how to avoid leading questions, how to stay neutral, how to probe deeper. These are important skills.
But I think one of the most important pieces of the puzzle gets overlooked: knowing what to do with the answer once you have it.
Research shows that even experienced interviewers ask leading questions. A study on qualitative interviewing found that leading questions appear in approximately 10-15% of interview exchanges, even among trained researchers. The difference isn’t whether leading questions happen, it’s how you handle them when they do.
From a research methodology perspective, the goal isn’t to fear imperfect questions. The goal is to recognise bias in your interpretation and adjust your analysis accordingly.
Why We Hear What We Want to Hear
My Kindle mistake wasn’t about question design. I didn’t ask a leading question. I asked “why do you like your Kindle?” which is perfectly reasonable. The problem was in what I did with her answer.
Research in cognitive psychology shows that we interpret ambiguous information through the lens of our own needs and expectations. This is called confirmation bias, but it’s more subtle than most people think. I wasn’t looking for evidence that Kindle was good, I was unconsciously translating her use cases into my use cases.
She said “you can read in the bathtub.” I heard “I read in the bathtub.” The words were the same. The meaning was completely different.
In HCI research, this is a well-documented problem. When users describe features or behaviours, researchers often map those descriptions onto their own mental models rather than the user’s actual experience. We fill in gaps with assumptions that feel reasonable but aren’t verified.
What Good Research Actually Looks Like
Here’s what I should have done, and what I teach teams to do in user research:
1. Probe for Specifics, Not Just Lists
When my friend listed Kindle benefits, I should have asked: “Which of these do you actually use?” or “Walk me through the last time you used your Kindle.”
Research on interview techniques shows that behavioral questions (what did you do?) produce more reliable data than hypothetical questions (what could you do?). When people describe possibilities, they’re often describing what sounds good rather than what they actually experience.
A classic study by Kahneman and Tversky demonstrated that people are unreliable at predicting their own future behaviour, let alone describing behaviour in the abstract. Ask about the last time, not about possibilities.
2. Distinguish Between “Can” and “Do”
This is critical. “You can do X” is a feature description. “I do X” is a usage pattern. They’re completely different pieces of information.
In product research, teams often confuse these. A user saying “I could use this feature for Y” doesn’t mean they will. It doesn’t even mean they’ve thought about it before you asked. It means it sounds plausible in the moment.
Research on stated versus revealed preferences shows significant gaps between what people say they’d do and what they actually do. Observational studies consistently find that self-reported behaviour overstates frequency of positive behaviors and understates negative ones.
3. Watch for Projection
When someone describes a use case that aligns perfectly with what you need, get suspicious. Not because they’re lying, but because you might be hearing what you want to hear.
I wanted bathtub reading to be a real use case because it solved my problem. So when she mentioned it, I latched onto it without checking if it was her actual behaviour or just a theoretical benefit.
Good research practice means tracking your own assumptions. Before analysing data, write down what you expect to find. Then explicitly look for evidence that contradicts those expectations.
4. Use Follow-Up Questions That Test Understanding
After getting an answer, paraphrase it back: “So you’re saying you personally read in the bathtub with your Kindle?” This gives the participant a chance to correct your interpretation.
Research on comprehension checking shows that this technique catches misunderstandings in 30-40% of cases where the researcher thought they understood but didn’t. It feels redundant, but it’s one of the most reliable ways to verify you’re capturing what someone actually means.
How to Analyse Data Without Bias
Even with perfect questions and good probing, interpretation is where bias creeps in. Here’s how to mitigate it:
1. Separate Observation from Inference
When taking notes or analysing transcripts, distinguish between what was said and what you think it means.
What was said: “You can read in the bathtub with a Kindle”
What I inferred: “She reads in the bathtub because she has a Kindle”
Keep these separate in your analysis. Review the actual words before drawing conclusions.
2. Look for Disconfirming Evidence
Actively search for data that contradicts your hypothesis. In my case, if I’d been thinking like a researcher, I would have noticed: she never said she personally does this, just that it’s possible.
Research on confirmation bias shows that people naturally seek evidence that supports their beliefs and ignore contradictory evidence. The fix isn’t to eliminate this tendency, it’s to build processes that force you to look for contradictions.
3. Involve Multiple Analysts
When possible, have someone else review your data and interpretations. They’ll catch things you miss because they don’t have your assumptions.
Studies on inter-rater reliability in qualitative research show that multiple coders catch significantly more nuance and identify more themes than single coders. It’s not about reaching perfect agreement—it’s about surfacing different interpretations.
4. Test Interpretations Against Behaviour
If you conclude “users want X,” look for behavioural evidence. Do they actually do X? Have they sought out X in other contexts? Or is X just something that sounds good when you ask about it?
The strongest research combines stated preferences (what people say) with revealed preferences (what people do). When these align, you have confidence. When they diverge, investigate why.
What This Means for Product Teams
This isn’t just about research methodology. It’s about how products get built.
How many features exist because someone interpreted “users could do this” as “users want this”? How many roadmap decisions are based on researchers hearing their own needs reflected back in user responses?
I’ve seen teams build features based on user interviews where participants described theoretical use cases that sounded compelling but never materialized in actual usage. The research wasn’t wrong—the interpretation was.
The most dangerous research isn’t poorly designed studies. It’s well-designed studies with biased interpretation that gives teams false confidence.
The Takeaway
Once I realised my friend didn’t actually do this, I went back and asked more specific questions about how she actually uses it. Turns out, the killer feature for her is being able to adjust text size and lay down in a comfortable position because she reads before bed.
Good research isn’t about asking perfect questions. It’s about recognising that every answer needs interpretation, and interpretation is where your assumptions take over if you’re not careful.
The next time you’re analysing user research (whether it’s interviews, surveys, or usability tests)ask yourself:
Am I hearing what they said, or am I hearing what I wanted them to say?
The answer matters more than you think.
If you’re designing research for your product: I work with teams to design studies that uncover real user needs, not just validate assumptions. Whether you’re running interviews, surveys, or usability tests, I can help you ask the right questions and, more importantly, interpret the answers without bias.
Research is non-negotiable. But the difference between research that drives good decisions and research that confirms what you already believed comes down to how you handle the answers you get.
🚀 Hi! I’m Andreea, an academic HCI researcher (gesture recognition and virtual reality interfaces) turned UX researcher that helps various businesses build products users actually love. I conducted hundreds of research studies for tech companies seeking clarity and I started this newsletter to share real examples and stories from my experience, so teams can do better research.
If you found this article helpful, I would be super grateful if you shared it with others.
I also love great discussions and debates, so I’m excited to hear your thoughts in the comments.
When I’m not doing research or writing, I like to go on hikes, read a good novel and play pretend with my dinosaur-obsessed toddler boy. If you’d like to support my work (and energy for writing, hehe), feel free to buy me a coffee!
Either way, thanks for reading and supporting my work ❤️
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.