Beyond the five-user assumption: benefits of increased sample sizes in usability testing

Area of Science:

• Human-Computer Interaction
• Usability Engineering
• User Experience Research

Background:

• The common practice in usability testing involves a small sample size, typically five participants.
• This approach is based on the assumption that this number is sufficient to identify most usability problems.
• However, the reliability and completeness of findings from small sample sizes remain a concern.

Purpose of the Study:

• To evaluate the risks associated with using only five participants for usability testing.
• To demonstrate the benefits of increasing the number of participants in usability studies.
• To quantify the impact of sample size on the percentage of usability problems identified.

Main Methods:

• A total of 60 users were recruited for usability testing.
• Random subsets of participants (5 or more) were systematically sampled from the complete dataset.
• The percentage of identified usability problems was analyzed for each subset size.

Main Results:

• Randomly selected sets of 5 participants varied significantly in problem detection, finding between 55% and 99% of all issues.
• Increasing the sample size to 10 participants raised the minimum percentage of problems found to 80%.
• With 20 participants, the lowest percentage of problems identified by any set reached 95%.

Conclusions:

• Relying on only five participants for usability testing can lead to incomplete problem identification.
• Larger sample sizes in usability studies provide more comprehensive and reliable insights.
• Recommendations for participant numbers in usability testing should consider the trade-off between cost and thoroughness.