Detectability of use errors in summative usability tests of medical devices: Impact of the test environment

The regulations on summative usability evaluations of medical devices (MDs) emphasize that the test environment must have sufficient ecological validity for generalization to real-life use. Here, we examined the influence of environmental fidelity (a component of ecological validity) on the detectability of MD use errors. A total of 140 participants participated in a summative usability evaluation of an anaphylactic shock auto-injector device under either a high-fidelity condition or a condition acceptable from the manufacturer's perspective, lower-fidelity condition. The numbers of errors detected in each condition were compared by applying descriptive statistics and logistic and Poisson multivariate regressions. We found that the level of fidelity did not influence the overall number of use errors detected but did influence the detection of certain use errors. To optimize the test environment and increase the detection of use errors, each environmental feature's role in the test task should first be examined.