Correlation of Narrative Evaluations to Clerkship Grades Using Statistical Sentiment Analysis

Area of Science:

• Medical Education
• Text Analysis
• Student Assessment

Background:

• Narrative evaluations are crucial for assessing medical students.
• Understanding the correlation between evaluation language and assigned grades is important for fair assessment.

Purpose of the Study:

• To evaluate the correlation between word choice in narrative clerkship evaluations and assigned letter grades.
• To introduce a novel text analysis method for evaluating narrative assessments.

Main Methods:

• Analyzed 100 clerkship evaluations (50 Family Medicine, 50 Internal Medicine) from 2020 OHSU graduates.
• Utilized textual sentiment analysis to determine collective sentiment and calculated average sentiment scores and character counts.
• Generated "word clouds" to visualize frequent sentiment word selections for Honors and Near-Honors evaluations.

Main Results:

• Sentiment scores positively correlated with assigned grades.
• No statistically significant difference in average sentiment scores or character length between Honors and Near-Honors evaluations within Family Medicine or Internal Medicine.
• Commonly used positive sentiment words included "outstanding" and "excellent".

Conclusions:

• Sentiment word choices do not significantly differ between Honors and Near-Honors clerkship narrative evaluations.
• The developed text analysis method can be adopted by other institutions for evaluating narrative assessments.