Fairness and objectivity of a multiple scenario objective structured clinical examination

Area of Science:

• Medical Education
• Clinical Skills Assessment

Background:

• Objective Structured Clinical Examinations (OSCE) aim for standardized, fair clinical skills assessment.
• Peer information exchange in OSCEs can lead to biased simulation management.
• A Multiple Scenario-OSCE (MS-OSCE) was developed to mitigate this bias.

Purpose of the Study:

• To assess if MS-OSCE, with varied scenarios per station, maintains objectivity and fairness.
• To evaluate the impact of scenario variation on student performance grades.

Main Methods:

• Five OSCE stations with 2-3 randomized underlying scenarios each were piloted.
• Student performance was evaluated using checklists and global ratings.
• Intraclass correlation coefficients analyzed the effect of scenarios and raters on grades.

Main Results:

• 169 students and 23 raters participated; Cronbach's alpha was 0.65.
• Scenario differences minimally impacted mean grades (0.03-0.4 on a 5-point scale).
• Scenarios accounted for 4-9% of grade variance, while raters accounted for 20-50%.

Conclusions:

• Scenario variation in MS-OSCE has a small effect on grades compared to rater variability.
• Enhancing rater training is more critical for ensuring MS-OSCE objectivity and fairness than standardizing scenarios.