Generalizability of Artificial Intelligence Assessments in Laparoscopic Surgery Simulation

The application of artificial intelligence (AI) in the assessment of procedural skills on a simulation platform using the global rating scale (GRS) has shown promise. Our team developed an open-source, low-cost simulation platform for the development of laparoscopic skills in low-resource settings, with skill assessment provided by video-based peer review and AI. The generalizability of AI trained on one procedure to evaluate general procedural skills within a single training system is unknown. This study examines the feasibility of generalizing AI-based assessments across procedures in a training system.

AI was trained, with varied combinations of procedures, to score 111 laparoscopic performance videos of four procedures (57 salpingostomies, 20 appendectomies, 15 enterectomies, and 19 diaphragmatic repairs), using time and distance-based calculations. Predicted scores were generated using five-fold cross-validation and K-nearest neighbors, with both 5-class (scored 1-5) and 2-class (pass/fail) scoring systems. Videos were also scored in a conventional fashion using human video-based review, based on GRS competencies.

AI assessments achieved 42%-100% concordance with human reviews in the 5-class system and 68%-100% in the 2-class system, P = 0.005. Within the 5-class system, 100% accuracy was reached when AI trained on multiple procedures evaluated appendectomy. The 2-class system attained 100% accuracy in three procedures across the GRS competencies.

AI assessment trained on procedures using video-based review evaluated laparoscopic skills across different procedures within a simulation-based training system. Dichotomizing scoring to pass/fail improved accuracy, while supporting the potential to assess procedural competence.