Validity of robotic simulation for high-stakes examination: a pilot study

Simulation is increasingly being used to train surgeons and access technical competency in robotic skills. The construct validity of using simulation performance for high-stakes examinations such as credentialing has not been studied appropriately. There are data on how simulation exercises can differentiate between novice and expert surgeons, but there are limited data to support their use for distinguishing intermediate from competent surgeons. Senior cardiothoracic trainees with limited robotic but significant laparoscopic experience ("intermediate surgeons", IS) and practicing robotic thoracic surgeons ("competent surgeons", CS) participating in a thoracic cadaver robotic course were evaluated on three Da Vinci (Xi) simulations. Scores were separately recorded into components and analyzed by t-test for significant differences between groups. 21 competent and 17 intermediate surgeons participated. Overall scores did not have a statistically significant difference in any exercise between groups. Simulation exercises do not appear to distinguish intermediate from competent surgeon performance of robotic skills. Without better validity data, the use of simulation for credentialing should be thoughtfully considered.