Relatively speaking: contrast effects influence assessors' scores and narrative feedback

Area of Science:

• Medical education research
• Cognitive psychology
• Behavioral economics

Background:

• Assessor scores can be biased by prior performances, a phenomenon known as contrast effects.
• Understanding the cognitive mechanisms behind assessor bias is crucial for improving evaluation accuracy.

Purpose of the Study:

• To investigate the mechanism and robustness of contrast effects in assessor judgments.
• To compare the influence of immediate versus a series of prior performances on assessor cognition.
• To determine if narrative comments are subject to similar contrast effects as numerical scores.

Main Methods:

• A blinded, internet-based experiment with 61 clinicians randomized into three groups.
• Participants viewed identical videos of varying performance levels (good, borderline, poor) in different orders.
• Assessor scores and narrative feedback were collected; comments were analyzed for valence and content.

Main Results:

• Contrast effects were observed after a single preceding performance, significantly altering ratings.
• The magnitude of contrast effects appeared to be influenced by an averaging of recent performance experiences.
• Narrative comment valence, but not content, exhibited contrast effects similar to numerical scores.

Conclusions:

• Judgments in assessment are inherently relative, aligning with findings in behavioral economics and psychology.
• Contrast effects extend beyond numerical scoring to influence the emotional tone of qualitative feedback.
• The practical impact of these effects on real-world assessment requires further contextual investigation.