Field testing when you don't know your customers goals


#1

Hi,

I started a new job on Monday it is a company that provides migration, reporting and management for cloud software. I have been tasked to look at the reporting tool. The problem is that while their customers find it useful, they don’t really know why. They have some ideas but need more. The co started out by providing a few reports and grew from there by adding things they thought customers would want and also things customers asked for.

I want to start talking to some customers and see how they use the platform. In a UX lab, I would normally write a script with tasks and record how the customer proceeds but as I don’t know what the tasks are, that wouldn’t work. I thought about remote mouse tracking but may have to sit through many hours t start getting any insights.

I considered a survey but that would probably lead to vague answers. I may do one but I need more.

Does anyone have any suggestions?


#2

Maybe not a survey, but start a conversation with, “What is your favorite report/result?” I don’t know what kind of research questions generally get asked. What I’m suggesting is a conversation with just a few customers to start getting an idea and then build a survey from there.


#3

Calling all @Researchers


#4

I like it the way @HAWK just summons to all researchers xD

it makes me remember of Naruto Anime. lol


#5

Hi Rachel,

It sounds like you have a lot of unknowns here. You need to get to know your users and understand how they solve their problems currently.

I would recommend user interviews and a contextual inquiry study. Start with general, get-to-know you questions, then get more specific. Ask lots of why’s around their behavior and motivation. Why is that information important to them? Why do they need to look at that data before this data? And once they answer, keep asking why until just before you think they’ll get annoyed. :nerd_face:

There are tons of articles on preparing for interviews. :smile:.

Take care,
Marie


#6

I tend to agree with Marie on this one. You may even want to interview one or two people from your company to understand what goals the reports are helping the business meet and how effective they are (if they are measured).

I would start by asking them what they find most useful. What goal/need does that help them meet? What else might they do to meet that need? You might then ask what is the next most useful to them and use the same follow up questions. This might seem repetitive, but it will likely get you some good insights into the situation.

You may also want to bring reports to the interview so participants can refer to them and show you what is useful to them if that is possible.

If it is more about HOW they use rather than WHY they use it, maybe have them navigate the system while you observe them (if that is appropriate for this project) and ask questions about their experience. Why do you do that? How is that helpful for you? How could this be better? Other questions that might matter.

I hope this helps. Good luck with your project!


#7

Some great ideas. Thanks, hopefully they’ll help to get me started. It is such a vague question, fortunately most of the other quetions I will probe are much less so.


#8

Look what happens when I do! ^^^


#9

4 posts were split to a new topic: Password support


#13

Do you have any analytics on those that are using it?
When I’m unsure about the reasons users are using something I always look at the extremes, usually is where I find the best insights.
So, in this case, you might have access to report usage statistics and you might find that some users have an above-average focus on the reporting tool, let’s say 80% of their activity over the past month has been generating reports, so they either found incredible value in this feature, have developed their own workarounds for repetitive tasks and if they are forced to use it and hate it, they probably have the list of annoyances written down already.
Now, on the other extreme, you will find the non-consumption crowd. Some users that use the app at the same frequency but completely avoid the reporting tool, they haven’t even opened it once. Talking to them it’s actually a great idea as well, you might understand their anxiety when it relates to reporting or things they are doing to replace it completely, offering a new perspective into increasing feature usage and satisfaction of users.


#14

I’ve written a script for a contextual study. My boss is in charge of strategy so will be able to organise talking to some of our users.

Should I post it here anonymized for feedback? It is my first contextual inquiry.


#15

Yes!


#16

I know I’d be happy to provide feedback!


#17

Hi,

Here it is Q is the company R is the main product I am researching and A is another product that we want to align R with.

Feel free to rip apart.

Contextual Study
**User & Consumer **
Set-up

The user should be in their natural environment or as close as possible to not disturb others or reveal confidential information

The user should have their usual device(s) ready though we will want to see how R***** is first accessed so we don’t want them to be already logged in unless they have the system permanently on and logged-in.
The difference between a contextual study and an interview is that we want to see the user in their own environment using the product to help direct the flow of the study.

The user should be advised that this exercise is an analysis of the R***** reporting tool and any other tools they use alongside with the aim of seeing how our users interact with it. They should be confi¬¬dent to make “mistakes” or use workarounds as we are judging the system not them.

It should be explained that the insights gained and any recordings if relevant will be used within the UI team and potentially shared with other managers or teams within Q***** to improve the system and that the data will not be made public.

prerecorded data

Name:
Company:
Years using Q****:

Questions/exercises

  1. What is your official job title?

  2. Beyond that, what are your primary responsibilities?

R----
3. Show me how you usually get to R*****?

  1. Does your work with R***** tend to be proactive or reactive?

proactive reactive both

If working proactively which reports do you generally use the most?

Show me how you get there?

How does the data inform your decisions?

What would be your next step?

If working reactively how do you usually become aware of an issue?

Which reports do you generally use the most?

Show me how you get there?

How does the data inform your decisions?

What would be your next step?

  1. Do you often look for new information, or do you tend to stick to the same set of reports?

  2. When you look for a new report do you usually find what you are looking for easily?

a. If no, Why?

b. Show me how.

  1. When looking for reports you use regularly do you usually find what you are looking for easily?

a. If no, Why?

b. Show me how.

  1. Do you use the search feature to search for reports?

Yes/No/__________

a. If no, Why?

  1. When you can’t find a report you want, what would usually be your next step?

  2. Do you use the scheduled reports feature?

Yes/No/__________

a. If no, Why?

b. Show me how?

  1. Do you save reports to access later from R*****?

a. If yes, do you have any annoyances?

  1. Do you tend to focus a lot on the visualisations on the dashboard or do you usually go straight for the raw data?

  2. Do you have any general annoyances with R*****?

  3. Are there any areas that you feel are lacking in R*****?

A----

  1. How do you access A*****?

  2. Do you tend to use A***** concurrently with R*****?

a. Can you show me an example?

  1. Do the reports in R***** help to inform your work with A*****?

  2. Do you record the changes you’ve made within A***** anywhere else?
    a. What do you record and how?

b. Do you use R***** to look retrospectively at changes you’ve made in A*****?

c. Can you give/show an example of where this has been useful?

d. Where there any follow-on actions because of this?

  1. When you make changes using A*****, are there additional external steps you must take that aren’t handled by A*****?

a. Do you have to send any communications to affected users?

b. Do you have to log data anywhere else such as in a spreadsheet, organisational chart, CRM, intranet etc?

  1. Do you use a mobile device for other work tasks?

  2. Would you use R***** or A***** on a mobile device if available?

  3. Are there any instances where alerts sent to your phone to alert you of particular events would be useful?

  4. Do you have any annoyances with A*****?

General
Do you have any observations that you would like to share about Q*****’s products?


#18

Hi Rachel,

Wanted to let you know I’m going through your interview guide today. I should have more for you tonight or tomorrow. Thanks!


#19

Hi Rachel,

Thanks for the chance to provide feedback! Overall, your questions are great! You’ll get good data from this, even if you don’t change anything.

Set-Up - The Set-Up section at the beginning sounds like instructions.

  • If it is, then I’d suggest adding something about taking note of the person’s surroundings, and any digital or physical artifacts that they interact with in the course of going through their processes.
  • Bring a 2nd person to take notes while you observe and interact with the person you are interviewing / observing. If you can’t have a 2nd person, ask the person involved if you can record audio in case you miss anything.
  • Another positive about having a 2nd person or an audio recording, you can play back and talk about what went well and what didn’t go well, so you can improve your interviewing / observation skills.

Ask WHY - Ask a lot of “why”s, “How so”s, and “Can you help me understand . . . ?”
You probably don’t need to write them down, unless you need reminders (as I do).
I noticed you started writing down “Why” alot in the last half of your questions for R***.

Observing Behavior trumps self-reporting - When you say “How does the data inform your decisions?” * - I’d encourage observing the behavior of when X happens, the person has to decide between A, B, or C, and ask “What information do you look at? Where do you find that information?” when they would choose between the options.

  • Watching their behavior - and asking questions about why they do what they do - will be much more revealing.
  • When you see that they are making a decision, ask questions like “Why did you go that way? What are the other options? And what information do you look at to make that determination? Where do you find that information?”
  • Side note, asking people how the data informs them can be useful if you want to understand what they think is the most important piece of information and what the variables are.
  • Asking the question gets a different answer than observing the behavior. What people say and what they do are not always aligned. People say what they have the best intentions for or what the perfect way of doing things is (the “happy path”).
  • Might need to reframe questions to focus on behavior rather than self-reporting.

Proactive vs Reactive reporting - Sounds like you would take a path to observe what they do proactively. And then come back and go down the path of reactive activities Good strategy!

  • You didn’t necessarily ask for this, but my mind went to how to analyze this data after you’ve collected it:
  • Compare and contrast the 2 paths
  • Depending on how important this observation is to your study, I’d even map out the process, and indicate those decision points, the reports run etc. Mapping might help in the comparison.
  • How many steps in each?
  • What reports are the same? What reports are unique?
  • Notice decision points. Why did they decide to use the reports they did?

”When you look for a new report do you usually find what you are looking for easily?”

  • The question could be reframed to not be leading (whether it’s easy or difficult) and not to be a yes/ no answer.
  • Ask the individual to show you how they pull new reports and observe their reactions.
  • If this study finds that findability is an issue to focus on, you might consider doing a card sorting or tree testing study afterward.

”Do you use the scheduled reports feature?”

  • Another way of reframing that question might be: When was the last time you used the scheduled reports feature? How often?
  • This line of questioning in your list made me think of another way of observing this behavior over time.
  • Do you have analytics in this reporting program? Or logs that Developers can look into from the back-end?
  • Is there some way to see how many of each report they pull?
  • You could show the data to the interviewee and ask why those reports are so important.
  • Why do is this report pulled so often?
  • Do you have any issues with pulling this report?
  • Can you show me . . ?
  • If no analytics or logs, then maybe a self-reporting survey with how often they pull the different reports, and in what combinations, would be a good alternative way of gathering similar data.
  • It’ll tell you what they all think are the most pulled or most important reports. Not the same thing as behavior, but sometimes it is what you need to do.

”Do you save reports?” “Do you use visualizations?” etc.

  • I would recommend reframing those questions to talk about the last time they accessed those features, why they accessed them, and ask them to walk you through using them.
  • And again, grabbing analytics (if you can) will show their actual behavior with those features, and then you can ask why they behaved the way they did. If that option is available to you.

I hope that was not too long to read! (lol) You can tell I totally geek out over this kind of thing.

Best of luck! Hope you get the insights you need!
~ Marie


#20

Wow thank you. That is amazing feedback. I will start going through it with a fine toothed comb.

I want to observe them using the reports but will have to see how that works as I feel the interviews may be a bit out of the flow of what they do. I will try to work it in.


#21

I’m glad you understood the split path for proactive/reactive. Some may be both too. That will be interesting to find out.

Since writing this I have been able to see in the stats that scheduled, saved and custom reports are really important to our users so will expand on that for certain.

Do you think the study may be a bit long for users?


#22

It might be a bit long. It’s hard to gauge length with an interview guide, because asking questions may lead you down a path that is important, and then you follow that path which may take you off in a different direction. If it seems important, I would go with that.

Because of that, I would prioritize what is most important and talking about what you really need to know about first, in case you run out of time.

Time box your interview to 45 - 50 minutes, and then wrap it up so you’re all done by 60 min max. Much longer than that, and you’re all brain-dead and the time isn’t as effective, I think.

And what you don’t get to, maybe you can discuss with other users in different sessions so you still get a chance to explore it?


#23

Hi Rachel,

Thanks for sharing your interview outline. I think it looks great. However, I had some observations and tips concerning some questions that might return unreliable answers. Mostly because of the way humans remember and reframe events in our lives (we are all deeply biased).

Question: Does your work with R tend to be proactive or reactive?

I think it will be somewhat tricky for the user to accurately describe this, by generalizing they might only remember new, recent details and reframe the answer to fit that.

Values, criteria, and behavior change when the user switches from a reactive to a proactive working mode and mentality, at the moment of the interview they might be biased, so it is better to get them into narration mode, where a current event can be described. Once the user starts narrating, it will be much easier to evaluate which actions and events the user rates as reactive or proactive, and most importantly what their needs are for each situation and how R fits into the experience.

This approach works best to uncover the needs that are currently satisfied by R, and maybe find unaddressed or hidden needs. It could be that one of those needs is satisfied when using A along with R and this could lead to discovering how to align both products.

Question: Do you often look for new information, or do you tend to stick to the same set of reports?

Do the reports have any date stamp? Could you have them go through a last set of reports and explain to you what the situation was like when they created each?

Maybe the user always accesses the same reports and only recently created a new one. However, for many reasons, the user isn’t sure if they should mention it, perhaps their report is incomplete, or they consider it not relevant. You might uncover attitudes related to the adoption of new reports.

Question: Do you use the scheduled reports feature?

Non-consumption of a feature might have powerful feelings associated with it. If you know what the need behind scheduled reports is, maybe start by figuring how strongly the user feels towards this specific need, and if they solve it currently whether they use an alternative way to do it and what is preventing them from switching to your solution.

In general, I’d avoid drilling too much into the product and features and instead understand the needs it is solving. If there are problems, how big are they? Where are the most significant opportunities to optimize the experience?

I’d ask questions like the following, just to give you an idea:

  • How were you solving this before R*****?
  • What made you look for a new solution?
  • What else did you consider?
  • If this particular report didn’t exist, how would you go about getting this information?
  • What is the most important thing when it comes to getting this information? (e.g., speed, accuracy)
  • Why is speed important to you for solving this need?

Hope this helps! Let me know if you have further questions.