Done or Almost Done? Improving OSCE Checklists to Better Capture Performance in Progress Tests
Debra Pugh1, Samantha Halman1, Isabelle Desjardins1
1a Department of Medicine , University of Ottawa, The Ottawa Hospital , Ottawa , Ontario , Canada.
Abstract:
Construct: The impact of using nonbinary checklists for scoring residents from different levels of training participating in objective structured clinical examination (OSCE) progress tests was explored.
Background:
OSCE progress tests typically employ similar rating instruments as traditional OSCEs. However, progress tests differ from other assessment modalities because learners from different stages of training participate in the same examination, which can pose challenges when deciding how to assign scores. In an attempt to better capture performance, nonbinary checklists were introduced in two OSCE progress tests. The purposes of this study were (a) to identify differences in the use of checklist options (e.g., done satisfactorily, attempted, or not done) by task type, (b) to analyze the impact of different scoring methods using nonbinary checklists for two OSCE progress tests (nonprocedural and procedural) for Internal Medicine residents, and (c) to determine which scoring method is better suited for a given task.
Approach:
A retrospective analysis examined differences in scores (n = 119) for two OSCE progress tests (procedural and nonprocedural). Scoring methods (hawk, dove, and hybrid) varied in stringency in how they awarded marks for nonbinary checklist items that were rated as done satisfactorily, attempted, or not done. Difficulty, reliability (internal consistency), item-total correlations and pass rates were compared for each OSCE using the three scoring methods.
Results:
Mean OSCE scores were highest using the dove method and lowest using the hawk method. The hawk method resulted in higher item-total correlations for most stations, but there were differences by task type. Overall score reliability calculated using the three methods did not differ significantly. Pass-fail status differed as a function of scoring methods and exam type, with the hawk and hybrid methods resulting in higher failure rates for the nonprocedural OSCE and the dove method resulting in a higher failure rate for the procedural OSCE.
Conclusion:
The use of different scoring methods for nonbinary OSCE checklists resulted in differences in mean scores and pass-fail status. The results varied with procedural and nonprocedural OSCEs.
Related Concept Videos
01:26Methods of Documentation II: POMR
01:21Methods of Documentation III: PIE
01:26Methods of Documentation IV: Focus Charting
It typically involves three columns for recording information:
01:20Nursing Process for Patient and Caregiver Teaching III: Evaluation and Documentation
Nurses can use several methods to evaluate patient outcomes. For example, oral questions can assess cognitive learning,...
01:15Nursing Evaluation
01:11Guidelines for Writing Outcome
Patient outcomes reflect the patient's response to the goal rather than what the nurse aims to achieve. Terminology should be observable and measurable to avoid the reader's interpretation. The desired outcome should be realistic and achievable in the designated care timeframe. Expected outcomes should align with adjunctive therapies. The outcome should enhance care...