Do outlier assessors provide useful narrative comments?

Outlier stringent/lenient and range-restricted assessors provide low utility assessment scores, reflecting assessor tendency more than learner performance. The quality of narrative comments generated by these assessors remains unknown.

End-of-shift assessments from an academic emergency department were scored using the Quality of Assessment of Learning (QuAL) score. The mean Qual score was calculated for each assessor. Stringency/leniency and range restriction were quantified using the mean-delta method and standard deviation of awarded scores respectively. Outlier vs. non-outlier assessor mean QuAL scores were compared using T-tests. Linear regression was conducted with mean QuAL score as the dependent variable, and assessor range-restriction as the independent variable.

2034 assessments were completed by 81 assessors. We identified 20 outlier assessors (10 stringent, 10 lenient). Assessor mean QuAL scores ranged from 1.2 to 5 (mean = 3.9, SD = 0.9). Mean QuAL score of non-outliers (4.0) was significantly higher than outlier lenient assessors (3.1, p = 0.02) but not different from outlier stringent assessors (3.8, p = 0.5). Range restriction was negatively correlated with mean QuAL scores (p = 0.003, R²=0.11).

Outlier lenient and range-restricted assessors provided lower-than-average quality narrative assessments. These assessors, where both numeric and narrative assessments are of limited utility, are ideal candidates for targeted interventions to improve assessment quality.