Natural language processing and entrustable professional activity text feedback in surgery: A machine learning model...

Area of Science:

• Medical Education
• Surgical Training
• Natural Language Processing

Background:

• Entrustable Professional Activities (EPAs) utilize narrative 'entrustment roadmaps' to define behaviors for different entrustment levels.
• Current roadmaps rely on expert consensus, lacking empirical data to guide entrustment levels.
• Analyzing assessment comments can enhance understanding of resident entrustment in practice.

Purpose of the Study:

• To explore the utility of natural language processing (NLP) for analyzing EPA assessment comments.
• To identify key behaviors associated with progressive resident entrustment and autonomy.
• To determine if NLP can provide data-driven insights for revising EPA entrustment maps.

Main Methods:

• Collected 1015 faculty EPA microassessments over 18 months from 64 faculty for 80 residents.
• Utilized Latent Dirichlet Allocation (LDA), a machine learning algorithm, to identify latent topics within EPA comments.
• Human raters reviewed LDA-identified topics for interpretability and relevance to entrustment levels.

Main Results:

• LDA analysis successfully identified topics that mapped directly to EPA entrustment levels (Gammas >0.99).
• Identified topics demonstrated a coherent trend with entrustment levels, with high-trust behaviors in high-trust topics and low-trust behaviors in low-trust topics.
• The analysis provided insights into specific behaviors influencing resident autonomy.

Conclusions:

• Latent Dirichlet Allocation (LDA) can effectively identify topics related to progressive surgical entrustment and autonomy from EPA comments.
• These data-driven insights can illuminate behaviors crucial for resident autonomy.
• Findings may facilitate evidence-based revisions of EPA entrustment maps for improved surgical training.