Tips for conducting cross-cultural usability testing?


#1

Hi all! I am new to this forum and relatively new to UX. I work for a company with a global market reach. While I have a strong background in research methodology and a lot of experiences in surveying international audiences (although mostly Europe-based), I am looking for any tips or guidelines that are specific to usability testing. I will be doing remote testing with respondents from different cultures (not just Western) and I am wondering if there is anything I should be particularly careful about. The fact that the interviews will be carried out in English will already have a stong effect on the data (quality), therefore I wouldn’t want to mess them up more with stuff that could have been controlled.


#2

Hi AnjaM! Welcome to the forums, and to UX. =)

I’ve not done any testing that deliberately worked with another culture. Other people might have more experience in this area?

I have done some work on projects that are pretty different contexts to my own. For example, I was working on a project that had a significant Japanese audience. I’ve never been to Japan or had the opportunity to properly understand their culture, but the sessions were in English, so I was taken aback when the first round of testing didn’t do very well; I was confused by the findings and didn’t get a good sense of mutual communication, despite good rapport and good participation in what I thought were clear expectations of speaking thoughts aloud, etc. I later realised that I was assuming the participants were explicit communicators (divulging information beyond the topic at hand based on the idea that the listener isn’t aware of background or correlating issues). Implicit communicators, such as the Japanese, tend to assume the opposite. Once I’d worked out my problem I was able to continue testing by doing some activities with participants beforehand set the scene differently and better encouraged them to share.

For cross-cultural testing, I assume some of the same principles would apply?
[LIST]
[]It’s much harder to understand context when more of it is alien to you
[
]Avoid stereotyping, which is easier to fall into when you’re floundering in new contexts.
[]Rapport and establishing a relationship will be more important than usual
[
]Addressing fears and concerns upfront will also be pretty important
[]Watch your assumptions. Suspend any judgements that aren’t specifically related to the research
[
]Make sure that you can clearly communicate the requirements to the participant
[]Be mindful of cultural taboos, inappropriate topics and the way race, gender and class might affect things
[
]Talking about previous activities and behaviour might be couched in modesty more than Western participants might exhibit
[]Verbal and non-verbal cues are likely to be different, so be mindful.
[
]Use straightforward language. Metaphors, proverbs and colloquialism’s are likely to confuse
[*]Be aware of the need for extra time - most people might translate the English into their native language to understand better before making a response.
[/LIST] It’s a mutual and collaborative process and so also goes the other way. Participants may be reluctant or make their own assumptions about intrusion based on their reading of you or your role as a researcher. Your attempts to avoid bias when commenting may also be more difficult.

Given the increased potential for misreading signals or behaviour, the need to have a second person scribing notes is even more important than usual. A second perspective might help pick up things you miss.

If you’re doing a lot of this, it sounds like it would be worth getting some language and cultural training so you can be competent and sensitive to a participants background?

Does the product you’re working on provide a translated version of the original content?

Does anyone else have experience in cross-cultural interviewing? Or anything that could be useful to AnjaM?


#3

Hi Lukcha! Thank you for your warm welcome and lengthy response!

The list of principles you provided is great, thank you – I did not consider some of them. Getting language and culture training would be a great idea, although my company (global publisher) literally is global, meaning that we have customers and users from everywhere – I will be doing interviews with people from all around the world, not just specific countries. Although learning a bit about each culture before interviewing somebody would only be helpful to me, so actually – why not.

Your example is very useful to me because we’ve just got results from a survey with a large Japanese sample back and the response rate was dismal, despite it being translated to Japanese by a native speaker. The main problem was probably asking them for opinions and criticism, and asking some questions they considered sensitive. Getting a second observer is a good idea, although probably not for all respondents – I would assume that some cultures would be much more affected by another person silently scribbling in the back of the room. I’m considering either a remote observer or somebody that would review the videos of the interviews later.

If you don’t mind me asking, what kind of activities did you do with participants beforehand to encourage them to share? Were these activities to make them relax more or to actually make them provide answers in a form that was more familiar to you?


#4

You’re very welcome, Anja. =) I’m glad to hear it’s being useful.

Having a second person along in your interviews is pretty standard practice, and the list of advantages in doing so is pretty long. There would need to be some strong reasons for not having their help, although audio recording and transcribing (and then reading - many hours) of the session may possibly help mitigate that. It just seems easier and cheaper to have a person there with you. But yep—getting someone to help you review and analyse the material would certainly be useful!

The activities I did beforehand to help set the scene with participants in that study did both—they helped define the rapport much better, as well as gave me the tools (not necessarily the insight) to understand their responses where I’d previously missed the signals. The activities were: [LIST=1]
[]I brought in another member of the research team who had the same cultural background. They helped me ensure the participant was clear about the purpose of the interview, and the kinds of topics we’d cover.
[
]​I also paid more attention than usual to addressing concerns and questions of the participant before we started.
[]After the first round of interviews didn’t go so well, I had quickly hand-drawn some index cards with some of these interview topics written on them, and we ‘played’ a sorting game where they put them in two piles, of either ‘private’ or ‘public’. Any cards in the ‘private’ pile that I could live without I didn’t ask, and the other ‘private’ ones I was careful to ask amongst the others during the interview (tended to be towards the end) only when I was sure I still had good rapport with the participant.
[
]We used a whiteboard to take turns playing a simple ‘game’ that acknowledged our different cultural backgrounds, and which also established areas of commonality between the participant and me that served as cultural ‘bridges’ later on (most of these were unique to each different participant) and started touching on the participants feelings about the topics on their ‘private’ cards and some of the other characteristics of the interview.
[/LIST] Looking back on it, I think being an ‘outsider’ ended up being an advantage, as many of the participants were able to talk about things in a way that I don’t believe they had with anyone from their own background.