Artificial intelligence classifies surgical technical skills in simulated laparoscopy: a pilot study

To develop a computer algorithm for the automatic classification of basic surgical skills in laparoscopy. The ability to objectively assess the operative skills of trainees would be invaluable for the success of competency-based medical education. Although technical advancements in computer vision have resulted in promising clinical applications, they have not yet been utilized in surgical education.

A single-institution, prospective study involving faculty and trainee surgeons recruited to use a bench-top simulator in order to complete the "precision cutting" task from the Fundamentals of Laparoscopic Surgery. An artificial intelligence (AI) computer algorithm was developed based on a transformer neural network model to classify videos of laparoscopic tasks as either executed by an expert or a novice surgeon. Performance metrics were reported in line with the Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis guidelines. The model was trained using fivefold cross-validation. The model's performance was evaluated using sensitivity, specificity, positive predictive value, negative predictive value, accuracy, F1 score, and area under the curve (AUC). The results were averaged across the folds, and 95% confidence intervals were computed for each metric. ROC curves were plotted to visualize the model's performance.

The internal dataset comprised 135 videos from 46 participants recruited between 2022 and 2023. Among these, 30 participants (65.2%) were junior surgical residents or medical students, and 16 (34.8%) were board-certified surgeons with prior laparoscopic experience. Following cross-validation, the AI model achieved an accuracy of 0.867 in classifying between novice and expert groups based on video analysis, independent of task completion time. For single-image classification, the model achieved an accuracy of 0.57.

This proof-of-concept study serves as a pilot investigation into the application of AI for classifying surgical skills, demonstrating the utility of computer vision in automatically and objectively classifying surgical expertise. While the results show promise, further validation is necessary to establish its utility in routine surgical training and certification. By providing objective evaluations, this technology could support and enhance the role of human evaluators in surgical education.