Generalizability of Artificial Intelligence Assessments in Laparoscopic Surgery Simulation

Area of Science:

• Medical simulation
• Surgical education
• Artificial intelligence in healthcare

Background:

• Artificial intelligence (AI) shows promise for assessing procedural skills on simulation platforms using the global rating scale (GRS).
• An open-source, low-cost simulation platform was developed for laparoscopic skills in low-resource settings, utilizing video-based peer review and AI for assessment.
• The generalizability of AI trained on one procedure to evaluate general procedural skills within a single training system remains unknown.

Purpose of the Study:

• To examine the feasibility of generalizing AI-based assessments across different laparoscopic procedures within a single training system.
• To evaluate the effectiveness of AI in assessing laparoscopic skills using a simulation platform.
• To compare AI-based skill assessment with traditional human video-based review.

Main Methods:

• AI was trained on 111 laparoscopic performance videos of four procedures (salpingostomies, appendectomies, enterectomies, diaphragmatic repairs) using time and distance calculations.
• Predicted scores were generated using five-fold cross-validation and K-nearest neighbors in both 5-class (1-5) and 2-class (pass/fail) scoring systems.
• Videos were also scored conventionally by human reviewers using GRS competencies.

Main Results:

• AI assessments achieved 42%-100% concordance with human reviews in the 5-class system and 68%-100% in the 2-class system (P = 0.005).
• 100% accuracy was reached in the 5-class system when AI trained on multiple procedures evaluated appendectomy.
• The 2-class system attained 100% accuracy in three procedures across GRS competencies.

Conclusions:

• AI assessment trained on various procedures can evaluate laparoscopic skills across different procedures within a simulation-based training system.
• Dichotomizing scores to a pass/fail system improved AI assessment accuracy.
• This approach supports the potential of AI for assessing procedural competence in surgical training.