Guerrilla test vs. traditional usability test & how do I record PHONE usability tests?

Process & Techniques
#1

Hi, community. Hope you are all doing well today!

I need a bit of advice. I am, as mentioned a few times by now, pretty new at this, so I am posting often to get feedback on my processes with a redesign of a cinema website that I intend to use as a portfolio piece.

Slowly but surely, I am getting very close to my first usability testing session. I have up a lot on usability testing, and I feel confident that I am ready to facilitate my very first usability testing session. But before I do that, I have a few more questions I keep coming back to.

Question 1: Should I be doing hour-long usability tests or go for a guerrilla test?
This one has been on my mind for a while now. I could opt for a traditional usability testing session, since I have a pretty good idea of exactly how to do it, where to find participants, etc., but if I did a guerrilla test, I could be taking the test to the actual cinema and ask people who arrive early for their movie if they can be bothered for 15 minutes. I know that finding participants of the criterea “orders movie tickets online” can easily be found elsewhere, but the guerrilla test has huge appeal because it seems like a rogue-ish and very proactive approach that fits the project well. I can always do a regular usability test for my next project. Also, less work involved. Right?

Question 2: How in the heck do I record this? [ANSWERED]
I want to test the website’s mobile version, and just the mobile version. I could, of course, rig up the test to be conducted on my own, personal phone, but then people would not be familiar with the common buttons and shortcuts, which would make the interaction clumsy and introduce a lot of platform-related usability issues. So how do I record this? My options seem to be:

  • Create a Brundleyfly rig (Steve Krug’s term for his little mobile-capture setup created from a cheap reading light and a webcam). This is my current bet, but I haven’t found a arm-clamp solution works for me yet.

  • Buy an expensive, professional doodad like Mr. Tappy (still in shock that anyone thought that was the best name for this gadget).

  • Get a friend to film over from an over the shoulder perspective.

  • Find a software solution that can record- and send me the interaction playback from any smartphone OS, and ask participants to install said software.

  • Make a static camera setup that I can bring with me, and then ask the participant to use her phone within a restricted physical space.

2 Likes
#2

Hi @DavidSkodt, good to hear that you’re ready to start testing. Let me start off with your first question:

Wrong question to ask :). You don’t select the approach and methodology in function of the tools that are available, you select your tools and approach in function of the research question you want to answer. So in this case it depends on what you want to figure out. Usually researchers tend to restrict usability tests to an hour, because of the effort that is being asked of participants. Guerilla testing is more of a style of conducting your test, and it not opposed to an hour long test. You probably want to compare a guerilla test vs. a lab test. I personally favour a guerilla test, because you can conduct the test in a more informal and natural setting. With a lab test you always have the simulation effect, which influences your participants a bit, and thus influences your results.

Usually the consideration is made based on both available resources and the insights you want to gain. My tip is, don’t overthink this process, just do it, and learn. Nobody nails it from the first time.

To record a session on mobile, I always used Lookback, which is a very good tool. I heard they removed the integration with Invision, which would be a pity if this is true. Also, ask yourself, why do you want to record this? Is it needed for your process? Do you want to convince stakeholders with raw footage? Given that this is a portfolio project, I say again, don’t overthink it. Don’t let this slow you down, or stop you. You can ask a friend to take notes on the fly, which saves you a lot of analysis time in the end (trust me, you don’t want to go through all the raw footage again).

My general advise is, since this is a portfolio project, adopt a hands-on approach, and just try out things. If your first round of tests didn’t go well (in terms of process), what’s stopping you from doing it again? Trial and error goes a long way, and your process is more valuable to your portfolio than the results.

6 Likes
#3

Thank you so much for your input, @glenn! I am immensely grateful for your feedback.

The distinction that I THOUGHT was implied was that the usability test would ask participants to sign on for the test ahead of time. In that scenario, it also seems much more reasonable to me to ask for more of their time. The guerrilla test, as I have understood it, is more of a “pounce in public” approach, where anything more than a 10-minute commitment seems like something I would decline, if I were being asked.

What I hope to find through my test is the most prominent usability issues of the website. The tasks I have written to find that out can be applied in both scenarios.

Do I need something like a hypothesis or research question to test for these flaws?

I want to record it so that I do not have to worry about notes during the actual test. I want to focus on moderating the test to the best of my ability. I am ok with watching the footage back and taking notes from that.
I want to apply rigorous documentation to my process, and I feel like I can make a much clearer case with documented interaction.
Also, without video, I cannot keep any quantitative data. Qualitative data is the focus here, but it would really be icing on the cake to have a measurable benchmark that I can compare my redesign to, for validation later in the process.

I do take this to heart! I really am not trying to be a perfectionist, I just want to make sure that I am not teaching myself bad habits.

2 Likes
#4

I agree with the advice Glenn shared as far as doing it to get out there and learn. Guerilla usability testing has been a way to start a research program at some places I have been, to convince them to do some research, as some research is better than none!

If you want to get a recording so you don’t miss anything, you can make a fairly inexpensive rig with the following items:

It has worked well for me. :slight_smile:

4 Likes
#5

Still trying to put a rig like this together! Havn’t found the right phone clamp and arm-part yet, being from Europe limits my selection.

2 Likes
#6

Not entirely incorrect, the 10 minute pounce in public is a good example of guerilla testing. So is setting up a “lab” in your nearest Starbucks :). I’m actually in favour of conducting user tests in “natural” environments rather than lab environments; in a lab the user will be more in a simulation mindset, because of the formal procedure and apparatus you would use (which could influence the results). But whatever approach allows you to reach your goal, cfr. identifying usability issues, is the one you go for. In your case, it’s probably not bad to have users sign on ahead of time, since you’ll want to use some of their time. That’s regardless of whether you test in a lab environment or somewhere else.

This question is a bit off as well, but don’t worry, we’re here to help :smile:. The reason this question is odd, is because in a correct process, you don’t define the research question/hypothesis before you’re ready to test. The user test is meant to validate your hypothesis. But if I understand your process correctly, you’re still in the discovery phase, so you’re establishing the problem? That means you’ll work towards shaping your hypothesis. You’ll identify issues with the current website, and by making a redesign of the website, you hypothesise that your proposal will improve the website and help accomplish the user their tasks more easily.

Make sure that you have explicit tasks, which you can validate in a binary manner (fail/success). You’ll make a redesign of the website, and you use the same tasks in a new test to validate your design proposal. You’ll have a very explicit and tangible validation of your hypothesis/redesign.

And as I said before, and I’m happy you take that advice to heart, just try and learn, don’t overthink these things too much. If I were to assess your portfolio, I would appreciate your process more than the end result ;).

Good luck with the tests, and keep us posted!

1 Like
#7

Yep :smiley:

2 Likes
#8

Really happy with the feedback, so far! I want to stress that I really appriciate being told where my understanding is misguided and why, so that I can learn and improve.

@glenn Can you give me an example of such a task? I am a little confused about the binary-part.
The way I intepret it, what you are saying is that I should write a task that can be failed, which just doesn’t seem to fit into anything I have been taught so far.

Also, can you go into a bit more detail (or provide me with some material) so that I can fully understand how a hypotheses fits into the UX design process? I have an academic background which has included plenty of research hypotheses, quantitative data and statistical validation, I just feel like some important link is missing in my understanding here.

As I mentioned, I do understand the importance of validation, which is why I want to make sure that I block the tasks of this test, and the test of the future redesign. With decent documentation, I should be able to validate through metrics like time spent and the amount of errors in a task.

#9

I’ve got a remedial question. What’s a guerilla test?

#10

A guerrilla testing session is an agile/lean approach to testing that usually takes place ‘in the field’. This means going to where people are like busy shopping centres and asking them to test a concept quickly (10 minutes at most). The upside is you can get insights quickly, the down side is it can take time to get a good amount of people to agree to be tested and they may not be your primary user group. I have done this in other situations like working on project for a Museum. We asked people as they arrived if they would have time at the end of their visit and we gave them a museum gift card by way of thanks. I have also done this in organisations when developing an intranet, this is why Guerrilla testing is also sometimes called Hallway testing.

3 Likes
#11

I don’t think using your phone for the testing would be a significant problem as long as it is the same OS. Also some users may not like you installing software on their phone. As glenn mentioned, lookback is a good app for recording. You can also try ‘DU Recorder’ which works well but only available for Android.

#12

+1 from me - I’ve used it testing an app for a bank and it was great

#13

Thank you!

#14

Everyone’s used to a different placement of the ‘back’ button and interaction scheme, I have seen friends and family really struggle to use my android-based smartphone if they are used to iOS. I am just worried that it will be an unnessesary distraction from the tasks. I could solve this with a preliminary “are you comfortable with using an android-based smartphone?”, but cutting away those who say “no” (considering they are the 1 in 10 who decided to give me their time of day) seems way too wasteful. Letting them use their own device is, in my novice opinion, essential.

I have looked into lookback before, it looks incredible for remote testing and it looks fairly easy and quick, but I still feel it is a little intrusive, as you also mentioned. No option is perfect, I know that, but the more I look into this, the more I feel like the “clip-on-rig” is the way to go.

I consider the “How do I record” part of this thread thoroughly answered, but please, feel free to add and join the conversation. Thanks for all the different inputs!

EDIT: Just found this universal phone holder.
I bet I could build a great setup with this, but the big clamp might be really awkward, getting too much in the way of how people hold their phones. I ordered it. I will let you guys know if it works or if I end up returning it.

2 Likes
#15

@glenn I don’t want to be invasive, but I am legitimately uncertain if you get notified if I EDIT in a @ in an existing post, so sorry if I just bumped you twice. Hope you find the time to get back to me on the questions posted further up the thread, I would really appriciate it. :slight_smile:

2 Likes
#16

Hey @DavidSkodt, I didn’t find the time yet to reply. I do get the notifications, and I’ll try to get back to you tonight, answering your questions :). Sorry for keeping you waiting.

3 Likes
#17

Such tasks could be:

  • log in to your account
  • select [x] as a payment method
  • order 3 tickets for yourself and 2 friends

It’s these kind of user tasks that you can validate quite easily, and measure your fail/success rate with.

1 Like
#18

Ah, now I get it. But can’t participants technically fail / succeed any task? Failing to reach the right screen or product, or expressing a desire to move on or end the test due to frustration?

1 Like
#19

Exactly, and formulating your questions/tasks in a certain way allows you to validate the test more easy, because you have more tangible success criteria.
In the end, your system helps users fulfil a certain need, in this case that they want to be entertained, by watching a movie. Your system allows them to buy tickets to a movie, that helps them fulfil that need. In order to do so, they need to accomplish a big task, buying movie tickets. To buy movie tickets, they have to go through a series of smaller tasks, such as selecting their movie, time, amount, and pay for the tickets.

You can break it down to that level (and you don’t explicitly have to ask them to select the amount of tickets they want, just tell them to get tickets for themselves and 3 friends for example).

If you haven’t conducted the test yet, feel free to share your testscript here. I can also share a pretty good link that might help you write your questions: https://www.nngroup.com/articles/better-usability-tasks/?lm=task-scenarios-usability-testing&pt=article

Hope this helps

2 Likes
#20

I am going to test tomorrow or friday. The scenarios I have written up are in danish, so some stuff might get lost in translation.

I asked my friends on Facebook (my friends are all pretty much between ages 20-34 and Danish. Practically everyone has used a smartphone for their entire adult life) what the 3 most common things they do on a cinema website is.

12 people replied.

X = number of times mentioned
10 What movies are showing
06 What times a certain movie is shown [on a particular date]
06 What movies are coming out
05 Order tickets
01 Ticket price
01 Opening hours
01 Cinema location

So I wrote up these scenarios. The stuff in [brackets] is just my headline for that scenario, but they are not shown to the parcitipant. I have iterated a few times on these tasks to ensure that none of the termanology matches the cinema site’s navigation. I’ll ask for 15 minutes of their time, which is just about what my pilot test indicates it will take.

[Exploratory]
Here is the site. First and foremost, just look around and tell me your impressions. What can you do on this site? What do they offer? What is the purpose of the site? Just look around and tell me what you think. You can scroll up and down, but please stay on this page for now.

[What movies are showing]
You feel like going to the cinema, but you can’t quite decide on a movie to watch. You decide to go to the cinema on tuesday, because you are free that evening. The nearest cinema to you is in the Field’s shopping center, in Copenhagen.

Find a way to see which movies are showing on tuesday.

[Reserve a ticket]
You and your two friends have discussed going to the cinema to see the movie “IT”. The only date on which you all can attend is friday the 8th of september, at around 8 pm. You decide to watch it at the cineam in Kennedy Arkaden, Aalborg.

Reserve three (3) tickets for the showing of “IT”.

[Which movies are coming out?]
You aren’t really interested in the current selection of movies showing. You decide that you want to check out what movies are coming out in october.

Figure out which movies are coming out in october.

1 Like