Correlation of Narrative Evaluations to Clerkship Grades Using Statistical Sentiment Analysis

Narrative evaluations are essential components of medical student assessment. This study evaluated how well narrative clerkship evaluation word choice correlated with an assigned letter grade.

One hundred clerkship evaluations, 50 from family medicine (FM) and 50 from internal medicine (IM), with even distribution of "Honors" and "Near-Honors" among medical students that graduated in 2020 from the Oregon Health and Science University (OHSU) were examined. A textual sentiment analysis, which evaluates positive and negative word choice, was used to determine each evaluation's collective sentiment. An average sentiment score and character count were calculated for Honors and Near-Honors evaluations from both clerkship disciplines. Sentiment word totals were used to form "word clouds" that highlight the most frequent word selections.

While sentiment scores positively correlated with the assigned grade, there was no statistically significant difference between the average sentiment scores among Honors and Near Honors graded evaluations within the FM or IM clerkship evaluation sets. There was no significant difference in evaluation character length among the assigned grades. Among FM evaluations, "outstanding" and "excellent" were the two most common sentiment words used in both Honors and Near-Honors. Among IM evaluations, outstanding and excellent were most commonly used in Honors evaluations, while "excellent" and "good" were most common in Near-Honors.

This study outlines a novel text analysis method for analyzing narrative evaluation association with assigned grade that other institutions can utilize. Sentiment word choices are not significantly different among Honors and Near Honors clerkship narrative evaluations.

The online version contains supplementary material available at 10.1007/s40670-022-01654-2.