Using artificial intelligence to uncover keywords associated with resident EPA entrustability

Entrustable Professional Activity (EPA) evaluations are the emerging foundation for providing resident feedback and assessing practice-readiness. No studies to date have utilized artificial intelligence to compare qualitative feedback with quantitative scoring.

This single-institution retrospective analysis applied natural language processing (NLP) models to elucidate keywords associated with resident entrustability. Resident EPA micro-assessments from 7/6/2023-12/3/2024 across post-graduate year levels were included. NLP algorithms and Spearman's correlations isolated keywords associated with entrustability and discrepancies between attending and resident-assigned scores.

One-thousand evaluations focused on gallbladder, intestinal disease, hernia, and acute severe illness cases. Lower entrustability themes included camera navigation, following the case, and retraction. Higher entrusted residents were described as "independent" and "safe." Attendings who rated residents higher than residents' self-ratings mentioned complex pathologies and emotional aspects.

This AI-based tool can be integrated across residency programs to identify characteristics of entrustable residents and help provide a valuable roadmap for development.