The reliability of six faculty members in identifying important OSCE items

Area of Science:

• Medical Education
• Clinical Assessment
• Psychometrics

Background:

• Objective Structured Clinical Examinations (OSCE) are widely used for evaluating clinical competence.
• Checklists are crucial tools for standardizing OSCE scoring, but their development and reliability can vary.
• Ensuring faculty agreement on the importance of checklist items is vital for accurate and consistent assessment.

Purpose of the Study:

• To quantify the level of agreement among faculty members regarding the importance of items on an OSCE checklist.
• To assess the impact of collaborative item selection on the reliability of OSCE checklists.

Main Methods:

• Six faculty members independently rated the importance of 47 items for a specific OSCE station (sore throat/hoarseness history taking).
• Inter-rater reliability was calculated using Cronbach's alpha for individual raters and the group mean.

Main Results:

• Faculty independently identified 15 out of 47 items as highly important.
• Individual rater reliability was fair (average alpha = .63).
• The reliability of the checklist based on the mean rating of the six faculty members was high (alpha = .91).

Conclusions:

• Collaborative development of OSCE checklists by a group of faculty significantly enhances checklist reliability.
• Group consensus on essential checklist items leads to a more dependable assessment tool compared to single-author checklists.
• This approach supports the validity and consistency of clinical skills evaluation in medical education.