Related Experiment Videos
Testing the raters: inter-rater reliability of standardized anaesthesia simulator performance
J H Devitt1, M M Kurrek, M M Cohen
1Department of Anaesthesia, Sunnybrook Health Science Centre, Toronto, Ontario. j.hugh_devitt@mail.magic.ca
Canadian Journal of Anaesthesia = Journal Canadien D'Anesthesie
|September 26, 1997
Summary
Physician performance assessment can be standardized using anaesthesia simulators. Observers showed excellent agreement when rating anaesthetist performance on videotaped simulation scenarios, indicating reliable evaluation potential.
Area of Science:
- Medical Simulation
- Anesthesiology
- Performance Evaluation
Background:
- Physician performance assessment has historically been subjective.
- Anaesthesia simulators offer potential for structured, standardized evaluations.
- The reliability of simulators for performance assessment is not well-established.
Purpose of the Study:
- To determine if observers agree on anaesthetist performance ratings from the same anaesthesia simulator event.
- To assess the reliability of using anaesthesia simulators for physician performance evaluation.
Main Methods:
- Two one-hour clinical anaesthesia scenarios with five problems each were developed.
- Video recordings of three role-played instances of each scenario were created (30 total events).
- Two independent anaesthetists scored the events using a defined rating scale, and inter-rater reliability was assessed using the kappa statistic.
Main Results:
- Observers achieved complete agreement on 29 out of 30 scored items.
- Excellent inter-rater reliability was demonstrated (kappa = 0.96, P < 0.001).
Conclusions:
- Videotaping anaesthesia simulation scenarios allows for reproducible scoring.
- Excellent inter-rater agreement was achieved in this study.
- Rating video recordings of anaesthetist performance in simulation settings is a reliable method for scoring.