Probing the effect of OSCE checklist length on inter-observer reliability and observer accuracy

Area of Science:

• Medical Education
• Clinical Assessment
• Emergency Medicine

Background:

• The Objective Structured Clinical Examination (OSCE) is a standard method for evaluating clinical skills.
• As OSCEs evolve, checklists may become more complex, potentially impacting assessment quality.
• Understanding the relationship between checklist length and assessment reliability is crucial.

Purpose of the Study:

• To investigate how OSCE checklist length influences inter-observer reliability and observer accuracy.
• To determine if increasing the number of items on an OSCE checklist affects the consistency and correctness of assessments.

Main Methods:

• Emergency physicians and residents assessed standardized video scenarios of OSCE stations.
• Participants used either 20-item or 40-item checklists for evaluation.
• A panel established 'gold standard' scores for comparison against observer ratings.

Main Results:

• Fifty-seven observers completed 228 checklists, with overall accuracy at 86%.
• Inter-rater reliability ranged from 58-78%.
• No statistically significant impact of checklist length (20 vs. 40 items) on observer accuracy or rating consistency was found.

Conclusions:

• Extending OSCE checklists by adding 20 items to a base of 20 does not diminish observer accuracy.
• Inter-rater reliability remains consistent regardless of whether a 20-item or 40-item checklist is used.
• Checklist length appears to be a non-factor in the reliability and accuracy of OSCE assessments.