Debate #2: Up-front User Research - Crucial or Not?


#1

With the success of Debate #1 still fresh in our minds, we’re kicking off number two, and this time we have fresh blood!

We’re lucky to have two skilled UXperts keen to take each other on over the subject of Up-front User Research.

So here is the low-down…

[SIZE=16px][B]The Topic[/B] [/SIZE]
Up-front user research is a crucial part of the UX process.

[B][SIZE=16px]The Contenders[/SIZE][/B]
Taking the affirmative position is Angela Schmeidel Randall, the founder of [URL=“http://www.normalmodes.com”]Normal Modes, a Texas-based user experience company. Angela’s superpower is working on experiences with large, mission critical datasets with complex rules. She speaks about cross-platform UX and the intersection of customer experience and user experience. Clients include Nike, Sandia National Laboratories, Avanade, Netflix, and Hearst. Fun fact: Angela’s first pet was a cow.

Taking the negative position is Donna Spencer, who runs [URL=“http://maadmob.com.au/”]Maadmob, an Australian freelance agency specialising in design strategy, user experience design and training. She recently read that Myers-Briggs ENTP types are great devils-advocates, for whom a good argument is more important than the actual content of the argument. That explained a lot.

So without further ado, I’ll throw it open to Angela to make her opening statement, and may the best woman win!
​Angela, the floor is yours…


#2

We’ve all heard the arguments for not doing user research at the beginning of a project:
[LIST]
[]It costs too much,
[
]It takes too much time,
[]We already know our users well enough,
[
]Our UX designers are so good that they can compensate for any gaps in user understanding, and
[*]If we have a problem, we’ll catch it at the end of the project during usability testing.
[/LIST]
There are four errors in this reasoning. First, [B]we underestimate the ROI of user research…if we calculate it at all.[/B] The cost and time arguments against user research are almost always straw men. Antagonists to user research use these arguments qualitatively, as if just claiming, “it’ll cost too much,” is a sufficient counterargument. But there’s an implicit comparison in this argument; you have to ask “Costs too much compared to what?” Maybe user research costs too much in comparison to a flawlessly executed development project, but when has that ever happened? Only by taking the time to “do the math” and calculate the ROI can you adequately assess the trade-offs in time and money and come to an informed decision.

Second,[B] we overestimate our knowledge of users[/B]. We think we understand their workflow, their domain-specific knowledge, and all of their wants, needs, goals and motivations. Maybe it’s because they gave us a comprehensive requirements specification. Maybe it’s because we inherited knowledge from other projects. Or maybe it’s just because we think we know better than the users. But the reality is that we can never really understand users unless we’ve worked in their shoes. Only through thorough observation of users performing their tasks can we discover and ultimately design the right solution for their needs. There are no shortcuts.

Third, [B]we overestimate our own skill. [/B]Have you heard of a cognitive bias called the Dunning-Kruger Effect? Basically, it says incompetent people are over-confident because they don’t know what they don’t know. These are Donald Rumsfeld’s famous “unknown unknowns.” If you’ve never watched the user performing the task at hand, no amount of experience or skill will enable you to imagine all the scenarios and cover all the bases. These can only be discovered through comprehensive user research.

Last, [B]we underestimate the cost of fixing problems later in the project.[/B] Financiers like to talk about the “time-value of money.” They talk about the “present value” of some future payment, and how the value of that payment today is substantially discounted over the duration. You can think of this as “How much do I need to save today in order to make that payment in the future?” We make the same intuitive calculation when we skip user research and push off problems to the future. We know that fixing problems in the future will cost something, but we never ask “How much is it worth to me today to address a problem that I’ll have to fix at some point down the road?” As a result, we usually grossly underestimate just how much fixing future problems will cost. A good rule of thumb is that the cost of fixing a problem increases by an order of magnitude at every phase of the development cycle. So a problem addressed in testing will cost 1000 times what it would have in the analysis phase.


Let me conclude with a real life example of what can happen when you don’t do upfront user research:

A Fortune 500 company was developing an enterprise application to manage its product line from concept to market. The project had failed user acceptance testing 3 times and costs had ballooned from $3M to $30M. The project team had grown toxic; high-ranking managers were focused on CYA activities instead of trying to resolve the problem—mostly because nobody knew what to do. The users were convinced the system would never work and they had low confidence in the technology group’s ability to ever meet their needs. They were even investigating purchasing a 3[SUP]rd[/SUP]-party solution. When we arrived, the technology group had just finished 3 weeks of 8-hour-day meetings with 15 project leaders and stakeholders in an effort to re-architect the project and ensure it’s success the fourth time around.

And yet, except for user acceptance testing, no one in the technology group had ever spoken with the users. An internal consulting group who claimed to “know” the end-users’ needs and workflow was the user representative to the technology group.

No user research had been done. The project requirements were based on nothing but heresay.

No wonder it failed. THREE TIMES. THIRTY MILLION DOLLARS. Wasted!

This is an expensive, especially egregious example, and obviously the organization had more serious problems than lack of user research. But it’s not uncommon. These are the kinds of turn-around projects we get pulled into as a result of no one doing up-front user research. They’re ugly. And they’re completely unnecessary.


#3

Whoa, nice start Angela! Donna clearly has her work cut out for her. I look forward to seeing her response.

It’s all yours Donna…


#4

Nice start, Angela! Wonderfully structured and argued. Agree with Hawk—Donna has her work cut out for her, indeed!


#5

Coincidentally Luke and I spoke to Donna on the phone today (about another matter) and she is very sick, poor thing! I’m sure she’s eager to rip into her opening statement, but we may need to be patient while she recovers … I’m sure it will be worth the wait!


#6

I’ve just heard from Donna, who is on the mend and will post her response on Monday. Thanks for your patience everyone.


#7

In true debating style, let’s first look at what the real crux of this issue is. I am not arguing that user research is not crucial, but that ‘up-front’ user research isn’t crucial - it’s the idea of ‘up-front’ research that is the point of this debate.

Angela has posted some very good arguments for why user research is important - that we underestimate its value, overestimate what we know about our users, overestimate our skill and underestimate the cost of fixing projects. These are all fantastic reasons to conduct user research, and I completely agree that by working with users we can save money and create a solution that’s great for the users and the business.

But none of the arguments explain why user research must be done ‘up-front’. Let me explain why I think that isn’t the best point for our research efforts.

Firstly, those of us who conduct research with people will have seen a common and interesting behaviour. When we talk to people about what they currently do, what they might need and what they might want, they almost always put these things in the context of what they know and work with already. They can tell us what is painful now and can offer tiny fixes to a current problem, but are generally not able to think broadly about their underlying needs, goals or what would make them happier or more efficient.

The normal answer given is that as good researchers we can listen and observe and extrapolate a solution based on what we do learn. But I believe that if we know that people can’t articulate what they need, it’s incredibly hard to get the right information to start extrapolating from.

Secondly, most of us are working on redesign work, or on projects that are starting from something that exists. Very few of us are working on problems that have never been tackled before. We might be redesigning a website or app, or working on projects for internal staff. In these situations, we [B]do[/B] already know a fair amount about our users - we can either learn from their existing behaviours (via informal methods, analytics etc) or we know about them because we encounter them day to day at work. I’m not suggesting we know all the nitty-gritty details that we’ll eventually need to, but that we already have some kind of foundation understanding.

So my premise is that we should allocate our time and our users’ time at a slightly different point. Instead of up-front research, it’s far more effective to make a start on a solution or design, and then get users involved. We can use what we know to get started - we don’t need a complete or thorough understanding of what people might need to do - we just need enough to make something up. It’s far more effective to show someone a design or explain a new service - people are great at reacting to what they see, imagining how they may use it, and especially to explaining why it won’t work for them.

Of course, there is a trick to doing this. You have to go into it with the right attitude. You can’t create a draft design and tell people how awesome it will be for them. You have to create something and take it to them with a humble approach of “We were thinking about this idea - how would that work for you?”.

No matter what, our time with users is always limited. We should use the time we have as effectively as possible, and my experience tells me that time is better used to discuss a rough draft of an idea than completely up-front.


#8

Ah ha! Nicely put Donna. This is shaping up to be a great debate indeed. You ladies are an awful lot nicer to each other than the boys were!

Does anyone have any comments on the arguments so far?


#9

Both arguments are brilliantly articulated and I agree with much from both.
I think the [B]crucial[/B] bit is that we absolutely need to know our users. How and when we achieve this understanding aligns to the words that should be printed on every UXers T-Shirt “It depends…”.

Having done this for a long while I’ve come to the conclusion there pretty much isn’t any silver bullet approach to a project or to research.
Some times up front is the right approach some time a ‘just in time’ or lean hypothesis based approach is the right choice.
The absolute for me is that I need research to understand the problems I need to solve. Indeed I need research to know if I have problems worth solving.

I think the Donna’s redesign argument falls down a little as often following existing perceived understanding doesn’t go to improve an existing site.
I’d actually say often redesign is where more upfront research is more valuable… almost every client or product owner I’ve worked with is entrenched in how they think users behave and often it’s not entirely true or accurate (again this depends). To get to a point where a site can innovate and potentially disrupt in my experience you have to disrupt existing thinking around a product or service. It’s a fast moving industry people and, even if research was done previously a health check, can’t hurt right?

One of the problem of designing with out research is that it can potentially propagate the entrench and inaccurate understanding

As for what users do and what they say being different I would say that there are multiple ways of conducting research that can help triangulate a deeper understanding.
Different methods are not mutually exclusive and I absolutely believe that testing hypothesis with prototypes (of pretty much any fidelity) is as valuable as other types of research.

Any way I’m off to get my T-Shirt printed.


#10

The t-shirt “It Depends. . .” is quite correct, in that there is no set rule for the best approach to a situation, nor is there any situation that will be a repeat of a previous project; it is almost a job requirement of a good UX designer to be in flux and malleable to any given situation they are presented. In some markets, there may be no precedent to work off of, and getting a bit of up-front research can go a long way to crafting a better end-product. . .but as Donna said, humans relate to things they are already familiar with. We make comparisons to create value, and that is a valuable assessment of a product’s ability to preform its goal successfully and smoothly. Without a working model to assess (which may not be available during initial research), there is no comparison to be made, and therefore no value to be given. On the other side of the spectrum, without attempting to glean some knowledge of the user base before creating a product, the potential to create a less-than-desirable model is much higher - that product will have been created from an attempt at reproducing already successful one. We as designers are required to bridge the gap between current market needs and past successes, which makes this career a constantly changing one (no cookie-cutter model will stay successful forever). Our past knowledge will be a fantastic starting point, but it is essential to be in touch with the ever-changing needs (and indeed desires) of the present.

This is a keystone of the debate, I think. Where is the designer coming into the picture, during the onset of the project, or during a redesign? User research is indeed valuable, but the investment and effort taken to optimize said research will be determined by the current stage of the project. I think it is agreed that retroactively changing a design model is more expensive than beginning fresh, and that getting it right (at least most of it right) at the onset is the best practice, so again. . .it depends.


#11

As a cofounder / developer creating something I don’t believe has been done before, I’m relating to Donna’s proposal that user research be deferred until there is something to test. This notion resonates with the Lean approach I’m cultivating, which is a remedy for my natural tendency to want to ‘get it right’ first time… a side effect of which is that I can spend too long in research. I’m drawn to the cycle of: deliver a minimum viable product, review, iterate.

If I picture myself on the golf course (unlikely), then excessive up front research seems like painstakingly plotting the perfect hole-in-one. I aim to leverage my experience to effectively get closer to the pin and recalibrate rapidly. Along the way I may find more attractive target, or identify a hazard which wasn’t obvious from the tee. By nature these shots are less likely to drop the ball in the trees, because I’m lobbing out into the obvious space ahead. I still have to aim, of course (I’m not attempting to use the force!) - to tee off without [I]any[/I] orientation would make for a long day walking.

It’s hard to disagree that upfront research is crucial, but the question for me is “[I]how much?”[/I]. If testing (producing an MVP however crude) is considered research, then I’m on board.


#12

I have just heard from Angela, who has also been unwell (it’s that time of year down under) but is working on her response today.


#13

[SIZE=15px]Thank you, Donna for your excellent rebuttal. I’m very much enjoying this exchange.[/SIZE]

[SIZE=15px]First off, let me say that any argument in favor of user research applies equally, if not moreso, to “up-front” user research. Once I dispatch with Donna’s arguments, I think you’ll see that it’s impossible to argue on the one hand that user research is “crucial,” but on the other that up-front user research is not. If user research is necessary at all, then it most definitely is best applied early in the development cycle, i.e. “the earlier the better.”[/SIZE]

[SIZE=15px]So let’s break down Donna’s argument. It comes in two parts: 1) for a new design, users are often unable to articulate their needs, and 2) for redesigns, designers often already have an intimate understanding of the users’ needs either through informal methods (like analytics) or through day-to-day communication. What is it about a redesign in her argument that suddenly endows users with the ability to articulate their needs escapes me, but I’ll move on.[/SIZE]

[SIZE=15px]Rather, I’ll argue that users’ inability to articulate their needs doesn’t obviate user research, it just makes the job of sussing out the user’s needs harder. But when user research is completed after the fact – instead of at the beginning of the project - UX designers are often already married to their vision of how users should complete a task. Any changes that come as a result of user testing are done in the context of the initial design work rather than based out of findings from structured conversations with users about their tasks, workflow, needs and constraints. When designers design before they listen, their subsequent efforts focus on making a square peg fit through the round hole they’ve created.[/SIZE]

[SIZE=15px]As for redesigns, they come in many different forms: those completed for various business purposes, those necessary for technical upgrades, and those completed simply because the existing product has usability concerns. Often it’s a mix of these three. In the case of a redesign that includes concerns about the existing application’s usability, it’s absolutely critical to conduct up-front research to prevent repeating mistakes of the past. [/SIZE]

[SIZE=15px]But regardless of the reason for the redesign, up front research is still necessary. Here are a few reasons why:[/SIZE]
[INDENT][B][SIZE=15px]Lack of continuity in redesign team. [/SIZE][/B][SIZE=15px]Between the original design and the new design, it’s very unlikely that the design team is composed entirely of the team that did the original design and user research.[/SIZE]

[B][SIZE=15px]Increased user experience with the product. [/SIZE][/B][SIZE=15px]The user has been using the product for many years, and they have more mature workflows and ideas about their needs, so their requirements may have changed.[/SIZE]

[B][SIZE=15px]Setting a new baseline to measure ROI.[/SIZE][/B][SIZE=15px] Presumably you followed best practices and did user testing on the old system before shipping it. Now the system as been in use for awhile; you may have even pushed some minor revisions into production. It’s a good practice to re-establish baselines for the UX against which you can measure ROI of future efforts. (Training baselines, how long it takes to perform processes, etc)[/SIZE]

[B][SIZE=15px]The most vocal users aren’t necessarily the most productive. [/SIZE][/B][SIZE=15px] The quiet wallflowers who sit in their cube and get their work done, are often not the ones reporting their needs. They weren’t the squeaky wheel, but they’ve got a wealth of insight to be shared. So basing your redesign on the communication you have with the most vocal users isn’t representative of your user base and may, in fact, miss feedback from the most productive users. [/SIZE]

[B][SIZE=15px]Analytics only sniff out questions to be asked of users[/SIZE][/B][SIZE=15px]. Analytics are great at telling us what we do right, but not at telling us what we’re doing wrong. They don’t provide the answers or insight into underlying challenges, motivations, and behaviors.[/SIZE]

[B][SIZE=15px]Skipping steps like user research adds risk and cost to the project budget. [/SIZE][/B][SIZE=15px]And it’s not a matter of if there will be additional costs, only how much those costs will be. No one wants to be in the position of reporting that, at the conclusion of user testing, the team found the application needs major revisions to launch successfully. Measure twice, cut once.[/SIZE][/INDENT]

[SIZE=15px]As I’ve laid out in this post and my previous post, user research is a best practice for numerous reasons. Designers who consider it optional, do so at their own peril. And again, I go back to the Dunning-Kruger effect — those who think they know enough about user behavior and assume they don’t need to conduct research are MORE likely to be wrong.[/SIZE]

[SIZE=15px]Before I conclude, I would like to thank Donna for a spirited debate and the folks at UX Mastery for providing this forum.[/SIZE]


#14

Great summation Angela. I’m really on the fence with this one.

I’ll invite Donna to rebut and then we’ll go to the polls.


#15

Of course in real life, the answer to everything is ‘it depends’, but this is a debate, where we get to think about things from a more extreme perspective and thus examine the principles upon which we make many of our decisions.

And because it’s a debate, Angela has attempted the traditional debating method of putting words in her opponent’s mouth :slight_smile: I did not say that users can’t articulate their needs during new designs - I said they can’t clearly articulate their needs ever. Nor did I say that designers have an intimate understanding of users’ needs for redesigns - I said that designers know a fair bit about users (I know a fair bit about my friends, but I’m not intimate with them!).

Apart from that misunderstanding, Angela’s main point appears to be that if you do a bit of design work to kick off your thinking and give users something to react to, that will marry you to your first design and lock you into a way of thinking.

In my experience, one of the characteristics of a good user experience designer is the ability to knock something together based on previous knowledge (of the product being designed, experience with similar problems or even just on business requirements) and their general UX experience. They can then use the draft design to start research - to get feedback, trigger discussion and explore concepts, and then to THROW IT AWAY. Good designers are good listeners, don’t get married to a design, are good at creating multiple approaches and comfortable iterating as they learn more. A good designer can absolutely do some initial design work, undertake user research and change the design based on it. To suggest otherwise is to suggest our entire field is made up of inexperienced, inflexible designers. That’s definitely not what I see.

Angela’s other points are all great reasons to conduct user research - the design team may not have been involved in earlier research, user needs have changed, we hear most from vocal users and analytics only tell part of the story. But they are still not reasons that research must be done up front - they are just reasons to do user research at some point.

To conclude, while user research is a key input to a design project, it is not critical to do it ‘up-front’. Because users cannot always articulate their needs, and because designers usually know something about the design and how it’s used, it can be far more effective to create an initial draft design. This draft design can be used to kick off user research - giving people something to react to, discuss and imagine how it would work for their task. Designers use the draft design plus the research to make changes (maybe even throwing the idea away entirely) and then do it all again. Ongoing research that evolves a design is much more effective than up-front research.


#16

I’ve loved this debate, and love that the debaters took some cues from Luke and I and began to get a few little personal digs in there. Bravo!

I’m on the fence on how to vote. Well done, you two! Some great points made on both sides.


#17

Make sure you jump into this thread and vote for the winner!