Optimizing assessors' mental workload in rater-based assessment: a critical narrative review

Area of Science:

• Medical Education
• Health Professions Education
• Assessment Science

Background:

• Rater-based assessments in health professions education impose significant cognitive demands on assessors.
• The quality of assessment ratings may be directly influenced by the mental workload experienced by assessors.
• Understanding and mitigating assessor mental workload is crucial for improving assessment outcomes.

Purpose of the Study:

• To review interventions and strategies for measuring and reducing mental workload in health professions education assessments.
• To explore the impact of mental workload interventions on assessment quality and outcomes.

Main Methods:

• A critical narrative review of English-language articles was performed.
• Searches were conducted in PubMed, EMBASE, and Google Scholar up to November 2018.
• Included studies reported interventions to measure or reduce mental workload in rater-based assessments.

Main Results:

• Six studies were included, all conducted in simulation settings (OSCEs or videotaped interactions).
• Objective measures did not show reduced mental workload after interventions like assessor training or reduced competency dimensions.
• However, reducing the number of competency dimensions assessed improved rating quality in three studies.

Conclusions:

• Mental workload in medical education assessment requires further investigation, including the development of valid measurement tools.
• Adjusting raters' focus appears to be a promising strategy for enhancing assessment outcomes.
• Future research should focus on optimizing assessment load to improve quality while balancing data requirements for judgment.