April 19, 2017
By Jessica Pichs, Principal, Research & Product Strategist
This question is a common one and in Ximedica’s experience, the answer is dependent on several variables.
Variable 1: What is the goal of your study?
At the initial stages of R&D, you may aim to understand who you user is and what unmet needs they have. In this context, you’ll want to interview about 6 to 8 users in each user group. (More about user groups when we get to Variable 2). Once early concepts and designs are starting to form and your goal is to gather feedback, it is advised to have these ideas evaluated frequently by the target user. A study at this stage need not be an elaborate undertaking - the input from 4 - 5 users (per group) can be invaluable for correcting or affirming direction.
As a project moves into engineering-heavy phases where design controls kick in and product specifications are being refined and validated, regulatory guidelines recommend that formative usability studies include 6-8 participants per user group. When the R&D team is satisfied that the design has been optimized and are ready for the final Summative/Human Factors Validation study to verify that all safety and efficacy mitigations have been resolved, regulatory guidance advises a minimum of 15 users per group.
There is grounding rationale for these numbers that may seem rather arbitrary. Researchers who have extensive experience in fieldwork will tell you that when observing and interviewing people, consistent patterns and themes of user behaviors and attitudes start to viscerally emerge at about the 6th to 8th interview mark. Moreover, empirical data from an investigation conducted in 2003 by Laura Faulkner evaluating the usability issues associated with a computer software program revealed that 15 users was sufficient to identify between 90-97% of the problems with the software.
In some cases, your goal may be to gather hard numerical data to drive a business case. So far we have only discussed qualitative research, but quantitative research that provides statistically significant numbers requires at least 30 (but ideally more) participants per user group.
Variable 2: Who is your target user population and how many divergent user groups does it include?
Invariably, there are bound to be several user groups who touch the product – perhaps there is a lay caregiver (a spouse, adult child or a friendly neighbor) who does not have the benefit of the disease education or regular exposure to the product. With a device used in a clinical environment, a doctor or nurse might be considered a primary user but inevitably there may be other users who have important interaction roles – perhaps preparing, assembling, storing, delivering and even cleaning or reprocessing – that impact the design, safety, efficacy and adoption of the device. Usability guidelines expect that the perspective of each different type of user be considered.
You may be lucky and only have one type of user – for example, patients aged 40-65 years with chronic heart disease. However, in Ximedica’s experience, few medical technologies function in this kind of simple universe. Another layer of complexity to factor in is when a defined user group has discernibly different needs, perspectives or limitations. For example, pediatric patients would need to be considered distinct from adult patients due to obvious cognitive and anthropometric differences. A user group that has great familiarity with a predicate product or extensive experience living with the disease state may need to be treated as separate from a novice or the newly diagnosed user, based on established mental models and expectations they bring to the interactions with a similar but new device. When designing your research, it is helpful to plan around relatively homogenous users who share the same goals, levels of training and physical capabilities to discern clear patterns and themes in the study findings.
Variable 3: How risk averse are you as a team?
Ximedica works with a wide range of companies and projects, that span across liberal and conservative outlooks, budget-rich and budget-strapped, highly aggressive and (sometimes, but rarely) generously scoped schedules. Each of these factors plays a role in the resources attributed to early development user studies. For example, when a project’s successful outcome represents a significant, game-changing risk for the business, the value of getting more input early and having more confidence in the direction increases exponentially.
Variable 4: How difficult is to identify and schedule time with the target user?
This last variable represents a conundrum that we often see at Ximedica. In a specialty clinical practice, there may only be a small group of medical practitioners with the experience you are seeking and these individuals are very, very busy people. Or perhaps the user is a patient with a rare orphan disease. Or perhaps the user is so ill that they are not able to spend sufficient time in a full usability evaluation. In these situations, hard decisions should be made on whether you really need the input from the actual user or whether a surrogate user group with the equivalent set of skills or capabilities will provide good guidance in a more efficient manner. If you are at the stage of preparing for your final Human Factors Validation study and you think you will not be able to get 15 users in each group, it is advisable to write to the FDA to determine whether having fewer participants is acceptable.
All 4 variables above should be considered as you plan your study and discuss the desired goals with the rest of the project team.