Human AI collaboration for unsupervised categorization of live surgical feedback.

Area of Science:

• Medical Education
• Artificial Intelligence in Surgery
• Surgical Skill Acquisition

Background:

• Formative verbal feedback is crucial for surgical training and skill development.
• Analyzing large volumes of surgical feedback is challenging due to data complexity.
• Current methods for feedback categorization are often manual and inefficient.

Purpose of the Study:

• To develop and validate a Human-AI Collaborative Refinement Process for discovering surgical feedback categories.
• To identify clinically relevant and actionable feedback themes from surgical transcripts.
• To assess the impact of AI-generated feedback categories on predicting trainee behavioral change.

Main Methods:

• Utilized unsupervised machine learning (Topic Modeling) on surgical transcripts.
• Incorporated human refinement to categorize discovered feedback topics.
• Correlated AI-generated feedback categories with trainee behavioral changes.

Main Results:

• Discovered feedback categories with high clinical clarity, such as "Handling and Positioning of (tissue)" and "(Tissue) Layer Depth Assessment and Correction [during tissue dissection]."
• AI-generated topics significantly improved predictions of trainee behavioral change compared to manual categorization.
• Specific feedback, like "Handling Bleeding," was demonstrably linked to improved trainee behavior.

Conclusions:

• The Human-AI Collaborative Refinement Process effectively identifies key surgical feedback categories at scale.
• AI analysis of surgical feedback offers insights beyond traditional methods, enhancing surgical training.
• This approach paves the way for automated feedback and cueing systems in live surgical environments.