Dimensions of ecological validity for usability evaluations in clinical settings

The development, evaluation, and eventual deployment of novel medical devices is a complex process involving various areas of expertise. Although the need for a User Centred Design approach to the development of both hardware and software has long been established, both current regulatory guidelines and widespread evaluation approaches fail to reflect the challenges encountered during day-to-day clinical practice. As such, the results from these evaluations may not provide a realistic account of the problems encountered by users when introduced to clinical practice. In this paper, we present a case study on designing the evaluation of a novel device to support laparoscopic liver surgery. Through a reflective account of the design of our usability evaluation, we identify and describe seven primary dimensions of ecological validity encountered in clinical usability evaluations. These dimensions are: 'user roles', 'environment', 'training', 'scenario', 'patient involvement', 'software', and 'hardware'. We analyse three recently published clinical usability evaluation articles to assess (and illustrate) the applicability and completeness of these dimensions. Finally, we discuss the compromises encountered during clinical usability evaluations and how to best report on these considerations. The framework presented here aims to further the agenda of ecologically valid evaluation practice, reflecting the constraints of medical practice.