A vision transformer for decoding surgeon activity from surgical videos.

Area of Science:

• Artificial Intelligence
• Robotic Surgery
• Surgical Skill Analysis

Background:

• Intraoperative surgical activity significantly impacts patient outcomes but remains poorly understood for most procedures.
• Detailed analysis of surgical actions from intraoperative videos is crucial for improving surgical quality and patient care.

Purpose of the Study:

• To develop and validate a machine learning system for decoding intraoperative surgical activity from robotic surgery videos.
• To assess the system's ability to identify surgical steps, actions, quality, and frame contributions.

Main Methods:

• Leveraged a vision transformer and supervised contrastive learning for analyzing surgical videos.
• Trained and tested the system on diverse datasets from multiple hospitals and continents.

Main Results:

• The system accurately identified surgical steps, surgeon actions, and action quality.
• Demonstrated generalization across different videos, surgeons, hospitals, and procedures.
• Successfully provided insights into surgical gestures and skills from unannotated videos.

Conclusions:

• Machine learning-based decoding of intraoperative activity offers a powerful tool for surgeon feedback and skill assessment.
• This technology can help identify optimal surgical behaviors and explore links between intraoperative factors and postoperative results.