Artificial intelligence for automatic and objective assessment of competencies in flexible bronchoscopy

Area of Science:

• Medical Education
• Artificial Intelligence in Medicine
• Respiratory Medicine

Background:

• Flexible bronchoscopy competence assessment traditionally relies on time-consuming, potentially biased expert raters.
• Objective, instant, and unbiased assessment methods are needed to improve training and evaluation.
• Artificial intelligence (AI) offers a potential solution for automated competency evaluation.

Purpose of the Study:

• To evaluate the validity of an AI-based system for automatic and objective assessment of flexible bronchoscopy competencies.
• To determine if AI can provide instant feedback on procedural skills.
• To compare AI-driven outcome measures with expert human ratings.

Main Methods:

• Participants performed simulated flexible bronchoscopies at the European Respiratory Society Annual Conference.
• An AI system assessed procedures based on diagnostic completeness (DC), structured progress (SP), procedure time (PT), and mean intersegmental time (MIT).
• AI ratings were compared with blinded expert assessments of anatomy and dexterity using a validated tool.

Main Results:

• All four AI outcome measures (DC, SP, PT, MIT) showed significant correlations with expert anatomy and dexterity ratings.
• Structured progress (SP) demonstrated stronger correlations with expert ratings than diagnostic completeness (DC).
• AI assessment showed objective and instant evaluation capabilities.

Conclusions:

• The study provides initial validity evidence for AI-based assessment of anatomical and navigational skills in flexible bronchoscopy.
• AI offers a promising tool for immediate, objective, and automated competency evaluation in bronchoscopy.
• Structured progress emerges as a key AI metric for assessing bronchoscopy performance.