Related Experiment Video
Updated: Jun 8, 2026

Evaluating Usability Aspects of a Mixed Reality Solution for Immersive Analytics in Industry 4.0 Scenarios
Published on: October 6, 2020
Estimating the number of usability problems
1Delft University of Technology, The Netherlands. h.kanis@tudelft.nl
Abstract:
The number of usability problems discovered in a user trial or identified in a heuristic evaluation can never be claimed to be exhaustive. This raises the question of how many usability problems remained undetected. In ergonomics/human factors research this subject matter is often addressed by asking how many participants are sufficient to discover a specific proportion of the usability problems. Current approaches to answer this question suffer from various biasing mechanisms, which undermine the credibility of the popular 'rule of thumb' that five participants are sufficient for the discovery of 80% of 'all' usability problems. This 5-user rule appears to be speculative in its application as a stop rule. In this paper, I compare actual estimates of the number of usability problems. Underestimation surfaces as a permanent threat. The so-called Turing estimate (C(T)) appears to be the most satisfactory. However, also C(T) estimates may suffer from underestimation. Therefore max(C(T),C(F)) with the C(F) estimate based on partitioned frequencies is proposed as the most adequate estimate of the number of usability problems in the studies presented.
Related Concept Videos
Estimation of the Physical Quantities
Problem-Solving
Testing a Claim about Population Proportion
There are two methods of testing a claim about a population proportion: (1) Using the sample proportion from the data where a binomial distribution is approximated to the normal distribution and (2) Using the binomial probabilities calculated from the data.
The first method uses normal distribution as an approximation to the binomial distribution. The requirements are as follows: sample size is large...
Accuracy and Errors in Hypothesis Testing
In hypothesis testing, the probability of making a Type I error, denoted as α, is commonly set at 0.05. This significance level indicates a 5% chance...
Documentation of Nursing Diagnosis
In some settings, data-driven computerized decision support systems are in place, allowing for more accurate nursing diagnoses. The database within one of these systems includes diagnostic labels defining characteristics, activities, and indicators for nursing. A nurse enters assessment...
What are Estimates?
The estimate for the mean of a sample is denoted by ͞x, whereas the mean of the population is designated as μ. Further, parameters such as the mean,...

