Using Natural Language Processing to Evaluate the Quality of Supervisor Narrative Comments in Competency-Based...

Area of Science:

• Medical Education
• Natural Language Processing
• Assessment Quality Evaluation

Background:

• Narrative assessment comments are crucial for learner development and promotion in medical education.
• Evaluating the quality of these comments is challenging and time-consuming.
• Existing tools like the Quality of Assessment for Learning (QuAL) tool exist but require significant manual effort.

Purpose of the Study:

• To develop and validate a natural language processing (NLP) model to automate the scoring of narrative supervisor comments using the QuAL tool.
• To assess the model's ability to predict overall QuAL scores and specific subscores.

Main Methods:

• Extracted and deidentified 2,500 Entrustable Professional Activities (EPA) assessments from emergency medicine residency programs.
• Had 50 faculty members and residents rate comments using the QuAL score.
• Developed and tested an NLP model to predict QuAL scores based on these human ratings.

Main Results:

• The NLP model achieved excellent performance, predicting human-rated QuAL scores within 1 point in 87% of cases.
• Subtasks related to suggestions for improvement and linking performance to suggestions showed high balanced accuracies (85% and 82%, respectively).
• Meaningful suggestions for improvement were identified as a key differentiator for high-quality feedback.

Conclusions:

• The developed NLP model can significantly reduce the time required to rate the quality of supervisor comments in medical education.
• This tool offers the potential for automated scoring of large volumes of assessment comments.
• The model can be utilized for real-time faculty feedback and to quantify/track assessment quality at various institutional levels.