Related Experiment Video
Updated: Jul 2, 2025
Evaluating Flight Performance and Eye Movement Patterns Using Virtual Reality Flight Simulator
Published on: May 19, 2023
Proficiency Levels and Validity Evidence for Scoring Metrics for a Virtual Reality and Inanimate Robotic Surgery
Juan C Tellez1, Imad Radi2, Rodrigo E Alterio3
1Medical School, University of Texas Southwestern Medical Center, Dallas, Texas.
Objective:
Our institution recently implemented a virtual reality (VR) skills curriculum for general surgery residents using the SimNow simulator. Based on a content alignment study, we revised the curriculum to include only 20 of 33 VR tasks and we added 3 previously validated inanimate tasks. The purpose of this study was to establish expert-derived proficiency levels for all tasks and to evaluate the validity of the scoring for the VR tasks.
Design:
Two expert robotic surgeons performed 5 repetitions of each VR and inanimate task. The trimmed mean (lowest scoring attempt and outliers [>2 standard deviations] were eliminated) was defined as the expert level for each task. For the VR tasks, expert levels were compared to resident performance to evaluate validity.
Setting:
This study was conducted at the University of Texas Southwestern Medical Center (Dallas, TX), a tertiary care academic teaching hospital.
Participants:
Two expert robotic surgeons participated in this study. The data from 42 residents (PGY2-4) who completed the original curriculum was used to represent novice performance.
Results:
Comparison of expert levels and resident performance was statistically significant for 15 VR tasks (supporting validity) and approached significance (p = 0.06, 0.09) for 2 VR tasks; expert levels were designated as proficiency levels for these 17 tasks. Group comparisons were clearly not significant (p = 0.2-0.8) for 3 VR tasks; 2 of these 3 tasks were retained as introductory exercises (with 3 repetitions required) and 1 was excluded. For the 3 inanimate tasks, expert levels minus 2 standard deviations were designated as proficiency levels.
Conclusions:
This analysis generated validity evidence for 15 VR tasks and established expert-derived proficiency levels for 17 VR tasks and 3 inanimate tasks. Our proposed curriculum now consists of 19 VR and 3 inanimate tasks using the selected proficiency levels. We anticipate that this design will maximize curriculum efficiency and effectiveness.
