Related Experiment Videos
The reliability of six faculty members in identifying important OSCE items
J Valentino1, M B Donnelly, D A Sloan
1Department of Surgery, University of Kentucky College of Medicine, Lexington, USA. jvale00@pop.uky.edu
Summary
Faculty consensus on checklist items for objective structured clinical examinations (OSCE) significantly improves grading reliability. Group-developed checklists are more dependable than single-author versions for assessing clinical skills.
Area of Science:
- Medical Education
- Clinical Assessment
- Psychometrics
Background:
- Objective Structured Clinical Examinations (OSCE) are widely used for evaluating clinical competence.
- Checklists are crucial tools for standardizing OSCE scoring, but their development and reliability can vary.
- Ensuring faculty agreement on the importance of checklist items is vital for accurate and consistent assessment.
Purpose of the Study:
- To quantify the level of agreement among faculty members regarding the importance of items on an OSCE checklist.
- To assess the impact of collaborative item selection on the reliability of OSCE checklists.
Main Methods:
- Six faculty members independently rated the importance of 47 items for a specific OSCE station (sore throat/hoarseness history taking).
- Inter-rater reliability was calculated using Cronbach's alpha for individual raters and the group mean.
Main Results:
- Faculty independently identified 15 out of 47 items as highly important.
- Individual rater reliability was fair (average alpha = .63).
- The reliability of the checklist based on the mean rating of the six faculty members was high (alpha = .91).
Conclusions:
- Collaborative development of OSCE checklists by a group of faculty significantly enhances checklist reliability.
- Group consensus on essential checklist items leads to a more dependable assessment tool compared to single-author checklists.
- This approach supports the validity and consistency of clinical skills evaluation in medical education.