Deep neural networks are effective tools for assessing performance during surgical training

Area of Science:

• Medical Education
• Artificial Intelligence
• Surgical Training

Background:

• Human evaluators are essential for assessing surgical performance in education and certification.
• Automated scoring using Deep Neural Network (DNN) techniques offers a potential solution for objective video performance assessment.

Purpose of the Study:

• To evaluate the accuracy of DNN models in classifying surgical performance from videos.
• To determine if DNNs can match human evaluator assessments in simulation-based surgical exercises.

Main Methods:

• Collected 254 videos of two simulation-based surgical exercises performed by attending surgeons.
• Scored videos by experienced instructors into three classes: expert, intermediate, and novice.
• Trained DNN models using Google Video Intelligence AutoML service on 2227 video clips.

Main Results:

• DNN models achieved 83.1% accuracy for the Ring & Rail exercise.
• DNN models achieved 80.8% accuracy for the Suture Sponge exercise.
• Models trained on individual exercises demonstrated over 80% accuracy in matching human classifications.

Conclusions:

• DNN models show high accuracy in matching human expert classifications for surgical performance.
• Automated scoring using DNNs may supplement or replace human evaluators in surgical education and assessment.
• This technology has the potential to enhance the objectivity and efficiency of surgical skill evaluation.