Natural Language Processing to Estimate Clinical Competency Committee Ratings

Area of Science:

• Medical Education
• Artificial Intelligence
• Natural Language Processing

Background:

• Clinical competency committee (CCC) meetings evaluate resident performance and inform learning plans.
• Faculty synthesize diverse data for holistic resident reviews.
• Natural language processing (NLP) offers potential for streamlining these assessments.

Purpose of the Study:

• To investigate the efficacy of NLP in predicting CCC ratings for surgical residents.
• To determine if NLP can aid in evaluating resident performance and development.

Main Methods:

• Analysis of end-of-rotation and CCC assessments for general surgery residents (2014-2018).
• Development of predictive models using assessment ratings and text data.
• Comparison of model performance with and without NLP-derived features to predict 16 Accreditation Council for Graduate Medical Education (ACGME) Milestones.

Main Results:

• Models incorporating both NLP and non-NLP predictors achieved the highest predictive accuracy (AUC = 0.87 ± 0.05).
• NLP-derived predictors alone showed comparable performance to non-NLP predictors.
• NLP effectively identified language correlated with specific ACGME Milestone ratings.

Conclusions:

• NLP is a viable tool for estimating resident performance ratings in preparation for CCC meetings.
• NLP can help faculty focus attention on residents requiring additional support.
• Automated text analysis can guide the development of targeted educational interventions.