Comparative validation of machine learning algorithms for surgical workflow and skill analysis with the HeiChole...

Area of Science:

• Computer Science
• Medical Imaging
• Robotics

Background:

• Cognitive surgical assistance systems require advanced surgical workflow and skill analysis.
• These systems aim to enhance patient safety through real-time warnings and robotic aid.
• Improved surgical training can be achieved via data-driven feedback.

Purpose of the Study:

• To investigate the generalizability of surgical phase recognition algorithms in a multicenter setting.
• To evaluate algorithms for more complex tasks: surgical action and skill recognition.
• To establish a benchmark dataset for validating AI in surgical analysis.

Main Methods:

• A multicenter dataset (HeiChole) of 33 laparoscopic cholecystectomy videos (22 hours) was created.
• Videos were annotated for surgical phases, actions, instruments, and skill levels.
• 12 research teams developed machine learning algorithms for the 2019 Endoscopic Vision challenge.

Main Results:

• Phase recognition F1-scores ranged from 23.9% to 67.7%.
• Instrument presence detection F1-scores were between 38.5% and 63.8%.
• Action recognition F1-scores were low (21.8%–23.3%), and skill assessment had an average error of 0.78.

Conclusions:

• Current machine learning algorithms show limited generalizability for surgical workflow and skill analysis.
• The HeiChole benchmark facilitates comparable evaluation of future AI models.
• Development of more open, high-quality datasets is crucial for advancing AI in surgery.