A multi-institutional study using artificial intelligence to provide reliable and fair feedback to surgeons.

Area of Science:

• Surgical Education
• Artificial Intelligence
• Medical Imaging Analysis

Background:

• Performance-based feedback accelerates surgical skill acquisition.
• Artificial intelligence (AI) systems analyze surgical videos for performance feedback.
• The reliability of AI-generated explanations for diverse surgeon groups is unknown.

Purpose of the Study:

• To systematically evaluate the reliability of AI explanations in surgical videos.
• To investigate potential biases in AI explanations across different surgeon sub-cohorts.
• To introduce and validate a novel method for enhancing AI explanation reliability.

Main Methods:

• Quantified AI explanation reliability by comparing with human expert explanations across three hospitals.
• Developed and implemented the training with explanations (TWIX) strategy using human explanations for AI supervision.
• Assessed AI performance and explanation bias mitigation through systematic comparison.

Main Results:

• AI explanations align with human experts but exhibit explanation bias, varying in reliability across surgeon groups (e.g., novices vs. experts).
• The TWIX strategy significantly enhances AI explanation reliability and mitigates explanation bias.
• TWIX improves AI system performance across different hospital settings and is applicable to current training environments.

Conclusions:

• Findings inform the integration of AI-augmented surgical training and credentialing.
• The study contributes to the safe and equitable advancement of surgical practice through AI.
• AI explanations, when refined, can support fair and effective surgical skill development.