Examiners' decision-making processes in observation-based clinical examinations

Area of Science:

• Medical Education
• Health Professions Education
• Assessment and Evaluation

Background:

• Objective Structured Clinical Examinations (OSCEs) are standard for assessing health professional students' clinical skills.
• Examiner judgment variability is a recognized factor influencing candidate scores in OSCEs.
• This study explores examiner decision-making processes and cognitive workload during high-stakes OSCEs.

Purpose of the Study:

• To characterize the cognitive processes and workload involved in examiner judgments of clinical performance in exit-level OSCEs.
• To identify factors influencing examiner decision-making, particularly for borderline candidates.
• To inform strategies for improving the accuracy and consistency of OSCE assessments.

Main Methods:

• Fifty-five examiners from five Australian medical schools participated.
• Examiners completed the NASA Task Load Index (TLX) to measure cognitive load.
• Post-OSCE focus group interviews explored decision-making for borderline and clear pass candidates, followed by thematic analysis.

Main Results:

• Examiners reported higher cognitive workload when assessing borderline candidates compared to clear pass candidates.
• Key themes in examiner decision-making included: marking criteria, adherence to marking sheets, candidate demeanor, patient safety, and calibration against a 'prototypical intern'.
• Mental demand was significantly higher for borderline assessments.

Conclusions:

• Judging candidate performance in OSCEs is cognitively complex, especially for borderline cases.
• Examiners use an intuitive construct of a 'prototypical graduate' when marking criteria are insufficient.
• Aligning this construct with marking criteria can improve examiner training, calibration, accuracy, consistency, and reduce cognitive workload.