Utilizing Natural Language Processing of Narrative Feedback to Develop a Predictive Model of Pre-Clerkship...

Area of Science:

• Medical Education
• Natural Language Processing
• Competency-Based Education

Background:

• The Feinberg School of Medicine uses a portfolio assessment system for reviewing narrative feedback for pre-clerkship learners.
• This review process is time-consuming and labor-intensive.
• Natural language processing (NLP) offers a potential solution for improving efficiency.

Purpose of the Study:

• To develop and validate a predictive model using NLP to assess pre-clerkship student performance from narrative feedback.
• To assist medical school competency committees in their review processes.

Main Methods:

• An iterative and inductive approach was used to analyze narrative feedback.
• Words and phrases were manually grouped into topics predictive of performance.
• Qualitative techniques, including member checking and iterative revision, were employed.

Main Results:

• Sixteen topic groups demonstrated predictive power for student performance.
• The optimal model combined topic groups, word counts, and categorical ratings.
• The model achieved an AUC of 0.92 on training data and 0.88 on test data.

Conclusions:

• A tailored NLP approach, incorporating qualitative methods, is crucial for analyzing medical education narrative feedback.
• Standard NLP packages are insufficient for predicting student outcomes in this context.
• The developed model provides a useful and salient tool for competency committee reviews.