Methods for Usability Testing an Educational Simulation: A Case-Study Using a Maternal Telehealth Scenario

Many women lack access to high-quality prenatal care and obstetrics services. We believe this gap can be addressed by training obstetricians and family practitioners how to use telemedicine to reach underserved citizens. Therefore, we developed a telehealth simulation to teach and measure resident physician telehealth competencies while managing a standardized patient with gestational diabetes. We designed a simulation usability evaluation to (1) identify simulation fidelity issues disrupting workflow, (2) ensure faculty could measure learner competencies using a scoring guide, and (3) assess simulation task difficulty. We modified a mixed-methods usability protocol for health technologies to measure task completion rates, perceived task difficulty, and faculty scoring guide reliability. We were able to measure seven of ten competencies. By contrast, three usability issues affected learner performance and limited the ability of faculty to measure three competencies. We are using this information to improve the simulation before piloting with residents. This report should be valuable to simulationists, educators, and human factors experts seeking to test educational simulations before piloting.