The Art of Usability Test Moderation

The reliability and validity of usability test data depends on the skill of the test moderator to navigate the complexities of managing human participants and managing their own (and their team’s) biases, expectations and judgements.

The data that comes out of a usability test will be used to drive product design or vet a product’s safety and usability. Effectively moderating a usability test is a critical part of your product development process and is both an art and a science.

A poorly managed participant can offer erroneous subjective or performance data as a result of unmitigated cognitive biases; similarly, a less skillful moderator might misinterpret a participant’s reactions based on their own biases. A moderator’s job is to understand the myriad variables at play during a usability study that can negatively impact the test outcome and lead to erroneous results: it means managing the participant’s emotional state (and their own!) and expertly navigating the test process like a Rockstar.

So, do you want to be a Rockstar moderator? With some hands-on practice, tips and guidance, you can master this art and science of moderation and ensure the data from your usability test is high-quality.

4 WAYS TO BE A ROCKSTAR MODERATOR:

1. DEVELOP RAPPORT QUICKLY AND EFFECTIVELY

This step is essential to collecting valid and authentic usability data. The more quickly and effectively rapport can be built with the participant, the more comfortable and open the participant will feel for the duration of the test session. The most effective way to build rapport is by being friendly and personable, not formal. Remind the participant that they are playing a big part in designing a medical device that will be in their world and have the potential to positively impact many lives. When the participant feels they are a member of the design team, rather than a user in a high-stakes test study, their nerves will settle, and their response and feedback will be more authentic.

Additionally, remind the participant that it is the device that is being tested, not them. Even though usability tests are designed to test the safety and effectiveness of a medical device through its usability, participants often blame themselves if they fail to complete a task or understand a device function or label. As a Rockstar moderator, remind the participant that any mistake they make actually helps identify areas where usability improvements may be necessary. This reminder helps to clarify the expectation of the participant in the test while highlighting the importance of their feedback – failures included!

2. REDIRECT THE PARTICIPANT AS NEEDED

The world of usability test moderation is full of contraindications that a Rockstar moderator must be able to balance. For instance, you must be able to work with participants when they get stuck but avoid coaching or “giving the answer away”. Knowing when it is time to either pause the participant and perform a root cause analysis or redirect them to get back on track successfully can significantly impact the results of your test session. Interjecting too early with root-cause analysis questions may prohibit the participant from figuring out a device functionality on their own, resulting in biased data for that test scenario. However, letting the participant struggle on their own for too long may negatively impact their confidence in themselves and the device, biasing the remainder of the test session. With practice, a Rockstar moderator finds the appropriate words to redirect the participant and the appropriate time to interject to ensure enough redirection is given without compromising the results of the test session.

Participants may also need redirection if their mind gets lost in design-world and their feedback is no longer aimed solely at the safety and effectiveness of the device pertaining to its usability. Or a participant may become tired and lose sight of the goal of the test session. As a Rockstar moderator, you must be able to empower participants to provide open feedback, while also keeping them on task and within scope of purpose of the study. Different types of participants require different moderation and redirection methods, so finding the balance between keeping a participant on track and on task while also giving them enough space for feedback can be tricky, but with enough practice, your participants will feel empowered and focused on the task at hand.

3. AVOID ASKING LEADING QUESTIONS

One of the most difficult tasks as a moderator is to not ask leading questions. The way you phrase or emphasize aspects of questions can impact the response of the participant. As a member of a device design team, you may feel emotionally invested in certain aspects of the design or have specific justifications behind user interface elements.

As a Rockstar moderator, it is critically important to not let these emotions bleed into your questions to the participant. Remember that your mental model does not reflect the device user’s mental model, and it is the user’s mental model that the device should be designed for. To more clearly understand the participant’s mental model, ask open-ended and neutrally-worded questions. For example, try following up with a single, unbiased question of “How did that go for you?” to uncover their thought process while not giving away any answers of how the test scenario was supposed to go.

4. PRACTICE “ACTIVE LISTENING”

When electing to use Think Aloud protocol in a usability test, a Rockstar moderator explains and teaches the think-aloud protocol to the participant so that the participant reveals their thoughts, feedback, struggles and motivations during the session. Then, it is time to practice active listening.

Active listening entails genuinely listening to and showing interest in what the participant is saying. Reinforce that you are actively listening through positive body language, eye contact, feedback and reflection. At times, active listening may also entail silence in order to avoid revealing correct or incorrect participant actions or answers.

While active listening, you are paying careful attention to the participant’s body language, verbal comments and task performance in order to obtain quality objective and subjective data. Rockstar moderators know that what participants do is often more important than what they say, and sometimes vice versa! By finetuning your active listening skills, you will be able to identify task failures or difficulties and also determine context-appropriate strategies for each participant and situation, overall resulting in higher-quality and more representative data.

SUMMARY

• Well-practiced moderation in a usability test can significantly impact the quality of the test data.
• Quickly developing rapport with the participant makes them feel more comfortable and confident, resulting in more accurate and authentic feedback/responses.
• Redirect the participant as needed to allow them to work through a use scenario on their own, pause them and perform a root-cause analysis, or re-focus their efforts on evaluating the safety and effectiveness of the device.
• Keep the moderation questions unbiased and open-ended, avoiding leading questions altogether.
• Encourage the participant to follow a think aloud process during the usability test.
• Practice active listening to obtain representative objective and subjective test data.

Whether you need help with compliance or designing intuitive user interfaces, Tensentric’s best in class Human Factors Engineering (HFE) services are designed to help you optimize the usability and use-safety of your products. We specialize in user research, risk analysis, user interface and rapid prototype design and evaluation, designing for CLIA Waiver, formative evaluations and HF validation testing following full COVID-19 precautions, regulatory support, and supporting studies that require BSL-1 or BSL-2 laboratory facilities. We also offer training in HFE processes to empower you to build the right product the first time.