Usability Testing Methodology

summary | detail | methodology

The usability testing process usually takes four to eight weeks, depending on the number of participants and the length of sessions, the ease or difficulty of recruiting, and the level of detail in the final report.

Test Design

The design of a usability test summarizes the issues of concern, the tasks to be observed, the questions to ask, and the criteria for screening the people who participate.

Tec-Ed meets with you to identify the issues to be explored in the test sessions, the characteristics of the desired participants, and the test tasks. Deciding which areas to test can be driven by current customer feedback, the product team’s concerns, or a prior expert (heuristic) evaluation. The following guidelines help us identify pertinent areas to be covered during usability testing:

Features and functionality that will be used by the greatest number of target users.
Elements that will cause the most problems if used incorrectly.
Features that generated debate or disagreements during the design process.
Elements where prior experience with related or predecessor systems leads us to suspect usability problems.
Features and functionality that you expect will require extensive user training.

Based on these discussions, Tec-Ed creates a test design document listing the goals of the research, the task scenarios, and the detailed questions and issues addressed by each task.

For a comparative study, the test design also describes the counterbalancing of product use. With counterbalancing, some participants try your product first, and others try the competitive product first. This approach minimizes learning effects that enable a user to perform a task better the second time around.

Test Materials

Following review of the test design, Tec-Ed creates a task script that the test facilitator uses so that all participants receive the same instructions and error remediation. We also prepare note-taking forms to speed data collection and analysis, as well as participant handouts and questionnaires as appropriate.

Test Sessions

Using the task script and other materials, Tec-Ed facilitates and observes the usability test sessions, taking detailed notes of the participants’ behavior and comments. Test sessions can last from 30 minutes to three hours; two hours is the maximum recommended length for uninterrupted test sessions.
Sessions conducted in Tec-Ed’s usability lab are digitally recorded using Morae software and videotaped as backup. Morae’s Remote Viewer allows the product team to observe the sessions in real time on their desktops; they view the participant’s computer screen and hear an audio track of the facilitator’s and participant’s voices.

Test Data Collection

Usability testing typically collects both behavioral and preference data. For the behavioral data, Tec-Ed observes how easily participants perform tasks and notes the problems they have. For the preference data, we observe participants’ reactions and collect their opinions about the interface, functionality, and workflow. We may also ask open-ended questions during a final questionnaire and debriefing interview.

The collected data has qualitative and quantitative components. For the qualitative data, Tec-Ed uses a combination of protocol analysis—also called “thinking aloud” protocol—and the observations of the usability researcher. (When thinking aloud, participants vocalize their thoughts and impressions as they perform tasks.) For the quantitative data, we record participants’ successes, problems, and “wrong turns” using the product. Often, we also administer Likert-scale questionnaires that enable us to quantify participants’ preferences, as shown in the following example.

It was easy to find the product I wanted to buy.

Strongly Agree	Agree	Somewhat Agree	Neutral	Somewhat Disagree	Disagree	Strongly Disagree
	√

Test Reporting

At the conclusion of the test sessions, Tec-Ed conducts a debriefing meeting with product team members who observed the sessions. That way, the team can begin work immediately on changes or enhancements everyone agrees are urgent.

Then Tec-Ed compiles and analyzes the collected data. Depending on the level of detail you want in the final report, the researcher may create a text database of notes (including participant quotes) for the issues explored in the usability test. Tec-Ed then prepares a results report describing our findings and recommendations. Report options include:

Oral presentations
Written reports with different levels of detail to meet your needs, from topline to in-depth results
Highlight videos

Report features may include:

Severity level assigned to each finding, based on a scale Tec-Ed co-defines with you.
Grouping of recommendations into short-term and long-term improvements.
Recommendations for design direction or for specific user interface improvements. (Tec-Ed also offers user interface design engagements.)

Test Participants

A key ingredient of successful usability testing is obtaining participants who are truly representative of the target users of a product. Research indicates that tests using four to six participants per “cell” (audience group/segment) can identify most of the usability problems likely to occur in large fractions of the total target audience.

Tec-Ed works with you to identify the desired number and balance of participant characteristics and summarizes the participant “profiles” in the test design. We also prepare a recruiting questionnaire, with quantitative screening criteria for each characteristic. Tec-Ed’s staff of experienced participant recruiters uses the questionnaire to screen and select participants who meet the agreed-upon criteria, regardless of geographic location.