Challenges in using rater judgements in medical education

Changes in the healthcare environment are putting increasing pressure on medical schools to make faculty accountable and to document the quality of the medical education they provide. Faculty's ratings of students' performances and students' ratings of faculty's teaching are important elements in these efforts to document educational quality. This article discusses selected research related to factors affecting raters' judgements, analyses how changes in the health care environment are influencing such judgements, offers some suggestions to moderate some of the effects and links these influences to the system that upholds professional standards. Ratings are known to have a positive bias (generosity error), provide limited discrimination and often fail to document serious deficits. The potential sources of these problems relate to the mechanics of the rating task, the system used to obtain ratings and factors affecting rater judgement. As managed care demands reduce the time faculty have for teaching, as system-wide disincentives to provide negative ratings proliferate and as social engineering challenges, such as the Americans with Disabilities Act, impose differential standards for students, the natural tendency to avoid giving negative ratings becomes even harder to resist. Ultimately, these forces compromise the capability of faculty to uphold the standards of the profession. The author calls for a national effort to stem the erosion of those standards.