Seeing the 'black box' differently: assessor cognition from three research perspectives

Area of Science:

• Medical Education
• Cognitive Psychology
• Assessment Science

Background:

• Workplace-based assessments (WBAs) are vital in medical education.
• Rater variability in WBAs is a persistent concern.
• Assessor cognition research investigates the cognitive processes influencing assessment judgments.

Purpose of the Study:

• To conceptually review different perspectives on assessor cognition in WBAs.
• To identify theoretical frameworks and literature informing assessor cognition research.
• To provide a nuanced understanding of variability in assessment judgments.

Main Methods:

• An international group of researchers engaged in iterative discussions.
• Key findings in assessor cognition research were shared and critiqued.
• A conceptual review of WBA assessor cognition perspectives was conducted.

Main Results:

• Three perspectives on assessor variability emerged: the assessor as trainable, fallible, and meaningfully idiosyncratic.
• The 'trainable' perspective attributes variability to incorrect application of criteria and inferences.
• The 'fallible' perspective highlights cognitive limitations and contextual influences.
• The 'idiosyncratic' perspective suggests differences may reflect legitimate, experience-based interpretations.

Conclusions:

• Each perspective offers insights but has limitations.
• A coexistent view integrating all three perspectives is proposed.
• This integrated approach aims to advance assessment quality and improve patient care.