Adjusting for Resident Rater Leniency or Severity Improves the Reliability of Routine Resident Evaluations of Faculty...

Area of Science:

• Medical Education
• Anesthesiology
• Statistical Analysis

Background:

• Accreditation Council for Graduate Medical Education (ACGME) mandates annual faculty performance evaluations.
• University-wide policies also require annual faculty reviews, making these high-stakes assessments.
• Reliability in these evaluations is critical to avoid Type I and Type II errors in performance assessment.

Purpose of the Study:

• To investigate the generalizability of findings regarding the unreliability of faculty performance evaluations when not adjusted for rater leniency/severity.
• To assess the impact of rater leniency/severity on the statistical reliability of anesthesiologist performance evaluations.
• To compare findings with previous studies from the University of Toronto and University of Iowa.

Main Methods:

• Utilized 11-item evaluations from the University of Florida during the 2018-19 academic year.
• Analyzed 3302 evaluations from 85 raters (resident physicians) assessing 108 ratees (anesthesiologists).
• Employed statistical methods including eta-squared, Shapiro-Wilk test, logit analysis, and intraclass correlation coefficient.

Main Results:

• Greater score heterogeneity was observed among raters (eta-squared 0.40) than among ratees (0.22), consistent with University of Toronto findings.
• Rater and ratee score distributions in the original scale were not normally distributed, similar to University of Iowa data.
• After adjusting for rater leniency/severity, 7 ratees were identified as significantly below average and 17 above average, contrasting with 0 in unadjusted analysis.

Conclusions:

• Routine faculty evaluations by resident physicians yield statistically unreliable results without adjustment for rater bias.
• Failure to adjust for rater leniency/severity leads to inaccurate performance categorizations.
• The necessity for rater adjustment appears to be a generalizable finding across multiple institutions for routine evaluations.