UX Researchers: How do you make sure you are asking users unbiased/insightful questions?


#1

I often find it’s a challenge to frame questions in the right way: whether surveying online with TypeForm, HotJar or in person interviews.

In cognitive psychology and behavioural economics, much research on framing effects (cf. Kahneman, Tversky) suggests that the wording of a question can greatly influence a responders answers.

Are there any tools to make sure I’m not asking biased or leading questions, or that the questions I’m asking my users are being answered truthfully?


#2

Don’t ask leading questions.


#3

Unfortunately, there’s no tool to use. It’s more of an art form.

Non-leading questions focus on, “what are your feelings/thoughts about…?”

If you know any psychologists or therapists, you could maybe ask them what they think about your questions. My reasoning is that those professions ask questions for a living.

I found this article that may be helpful.


#4

It is true that the wording of a question can greatly influence the respondent’s answers.

There is bias baked into all survey and interview design, you can’t hope to escape it but you can hope to account for it in a way that lets you separate signal from noise. As long as you are onsistent your approach, you’ll learn something from your data - even if it is just “My interview script resulted in too many hypothetical answers!”

For interviews, this is a good primer here and the last time I did an interview series I followed this general approach>

  • No script, just had a list of topics/ground to cover and then segued between them in any direction to suit a more natural conversation
  • Directed those conversations towards real-life, recent incidents: “tell me about the last time you…” and avoiding “what do you typically do when…” generalizations
  • Questioned everything participant stated to be important/valuable/necessary, and diplomatically: “But why is that important?” Kinda relates to the 5 why’s thing. Also JTBD-style is a great model.
  • I tried to let there be some negative space in the conversation…it can be difficult to resist the urge to “help them out” with leading answers at first but it’s better to have an awkward silence

As for surveys, there are lots of cognitive biases to fight there so it just depends on what your survey’s objective is. These were the principles I used in the last survey I constructed:

  • Consistent ratings scales within a survey
  • Minimize order bias by randomizing the choice or question order
  • Blending a balanced mix of negative and positive phrasing
  • Keep longer questions with context more neutral, indifferent and free of editorializing

#5

I do this a lot. Depending on what I’m doing, I’ll often have a paragraph or two at top to ensure I’m giving information consistently to all participants and then just have a checklist below. As the conversation naturally progresses, I just tick things off my list and then when it runs out of steam, I ask questions to get what’s missing. It helps me too because I’m autistic and have a hard time following the flow of a conversation so it helps me focus and keep track of what’s already been discussed. I used to ask the same question twice because I forgot I’d already asked it- super embarrassing!

Oh this is fun! A workshop facilitator once told me that most people cannot stand silence for more than 11-15 seconds without feeling the urge to fill it. You have to wait them out and let them fill it first. I find it really easy because I’m really comfortable with silence (again thanks to my neurodiversity) but I recognise that it can be really hard.

Yes! Like tell me a story…
No future predictions either! Avoid asking things like “what would you do if this was released tomorrow?” I don’t know? Eat a sandwich? :sandwich:


#6

There are a lot of good answers here. I do topics with suggested questions underneath (in case I draw a blank). I really look for stories, but it takes people some time to warm up.
Always focus on a specific time period when asking about using something. Try to get their process. Another thing I do is work backwards. If I got the most perfect information, what would that look like? Then how to structure the question or questions to hear those stories.
I very rarely ask preference questions as I feel like those are leading. Never ask anyone to imagine something. Never ask if the participant or someone else would use something.

Also, always run your questions by one or two people to see what they say.

Sadly, no tool, but we do have the community of UX people. :slight_smile: