Pass-Fail Decisions for Borderline Performers After a Summative Objective Structured Clinical Examination

Area of Science:

• Pharmacy education
• Medical education assessment

Background:

• Objective structured clinical examinations (OSCE) are crucial for assessing pharmacy students' clinical competency.
• Standard-setting methods are used to determine pass-fail criteria, but expert judgment also plays a role.

Purpose of the Study:

• To identify criteria expert assessors prioritize when making pass-fail decisions for pharmacy students using OSCE data.
• To evaluate the inter-rater reliability of these expert judgments.

Main Methods:

• Three expert assessors reviewed summative OSCE data for eight borderline pharmacy students.
• Assessors made pass-fail decisions and were interviewed to elucidate their rationale.
• Inter-rater reliability was calculated using intraclass correlation coefficients.

Main Results:

• Expert consensus was reached for only three of eight students, with decisions diverging from standard-setting outcomes.
• The reliability between assessors' judgments was poor.
• Assessors prioritized task-focused competencies (e.g., recommendations) over global ones (e.g., communication skills), despite equal weighting in scoring.

Conclusions:

• Expert assessors evaluating borderline pharmacy students on OSCEs focus on similar competencies but differ in performance expectations.
• Pass-fail decisions are predominantly influenced by task-specific skills rather than holistic evaluations like communication.
• The study highlights a discrepancy between scoring weights and assessor emphasis in pharmacy OSCE evaluations.