A Question of Bias
by Laurie Kantner
Whenever we study someone using a product, website, or service, the behavior we observe includes the facilitator’s effect on, and perception of, the events. This interaction can skew, or bias, the data collected and may produce unreliable results—that is, results that vary when a different person conducts the same study.
Acknowledging that facilitation introduces bias is not to devalue human-facilitated user research. Although bias is unavoidable, it is controllable, and the results of human-facilitated user research are often deeply insightful. The unprofessional response to a concern about bias would be to deny the possibility of bias and thus not take measures to minimize it or at least identify where it may have a strong effect.
User research professionals understand well-established rules to minimize the facilitator’s effect: maintaining a neutral demeanor (tone, body language), asking open-ended questions, avoiding “loaded words” and “leading the witness,” and so forth. Some practitioners need reminding that “UI speak” terminology in questions and probing—words such as “navigation” and “radio buttons”—can confuse participants or narrow their focus to widgets instead of the goals and tasks we want to study.
Although these are obvious ways bias is introduced by inexperienced researchers, a way that is less obvious to people outside the user experience field is when designers, who may not be trained researchers, conduct usability evaluations of their own work.
Designers follow disciplined processes for understanding user requirements and create usable interfaces that suit those requirements. As with any creative profession, experienced designers know they must separate their ego from their work, and actively seek feedback to improve their designs. They may even delight in unexpected user behavior. They may wish to have an independent researcher conduct a usability evaluation of their work, yet resort to doing it themselves because budget or resources are lacking, or because they perceive that another person will have insufficient domain knowledge to conduct the evaluation.
Yet, to prepare for usability sessions means developing a sufficiently detailed moderator’s guide to ensure appropriate task coverage, consistency, neutral everyday language, and open-ended probing. The designer must stop designing and start planning the evaluation. A designer may instead fill the available time continuing to improve and refine the design (so there is less to catch during usability evaluation) or creating design alternatives, and run short of the necessary time to create tools to ensure unbiased facilitation.
During sessions, it is difficult to watch a user stumbling over one’s design without thinking about ways to improve it. These thoughts compete with paying attention to user behavior. If the deadline to make design changes is soon, then shortcuts—such as limited probing, closed-ended questions, and discounting specific user comments—look inviting as ways to narrow the focus of potential revision work within the available time. A designer untrained in research may also be inclined to rush to a quick solution instead of exploring deeply why the user is stumbling.
A separate, independent researcher feels less attached to a particular design solution, and collaborates with the design team to choose appropriate tasks and identify suitable test participants. This researcher creates a detailed script and shows it to a peer researcher to ensure it uses no leading questions and insider vocabulary. A separate researcher will plan time to “rehearse” the session to ensure smooth and consistent delivery.
Engaging a separate researcher for usability evaluation is practical when a team plans from the start and saves budget for it. A strong researcher will learn the details of the domain and the issues setting the context for the evaluation. A team can mitigate the bias that naturally occurs when a designer evaluates his or her own work by assigning a separate researcher, collecting reliable data through objectivity and rigor.