Related Experiment Video
Updated: Aug 15, 2026
10:32
Development of a Virtual Reality Assessment of Everyday Living Skills
Published on: April 23, 2014
Assessment of clinical performance during simulated crises using both technical and behavioral ratings
D M Gaba1, S K Howard, B Flanagan
1Anesthesiology Service, VA Palo Alto Health Care System, California 94304, USA. gaba@leland.stanford.edu
Anesthesiology
|July 17, 1998
Summary
Anesthesiologist performance in critical events can be assessed using patient simulators and video recordings. While technical skills were rated highly, behavioral assessments showed variability, indicating a need for refinement in rating systems for educational use.
Area of Science:
- Medical Simulation
- Anesthesiology
- Performance Assessment
Background:
- Effective assessment of anesthesiologist performance during critical events is crucial.
- Patient simulators offer a standardized method for presenting crisis scenarios to clinicians.
- This study investigated performance ratings and interrater variability using videotapes of simulated crises.
Purpose of the Study:
- To evaluate the reliability and validity of performance ratings derived from simulated critical events.
- To assess both technical skills and crisis management behaviors of anesthesiology teams.
- To determine the interrater variability in performance assessments.
Main Methods:
- 14 anesthesiology teams managed two simulated crisis scenarios: malignant hyperthermia and cardiac arrest.
- Technical performance was rated using scenario-specific checklists (0.0-1.0 scale).
- Crisis management behaviors were rated on a five-point ordinal scale, with statistical analyses for interrater variability.
Main Results:
- High technical performance ratings were observed (MH: 0.78, Cardiac Arrest: 0.83).
- Crisis management behavior ratings varied, with some teams performing at minimally acceptable or poor levels (28% MH, 14% Cardiac Arrest).
- Interrater agreement ranged from fair to excellent, depending on the specific item and statistical test employed.
Conclusions:
- Videotapes of simulations effectively allow assessment of both technical and behavioral performance.
- The current behavioral rating system requires improvement, particularly in aggregating ratings for behaviors that change over time.
- Performance assessment tools show potential for educational research and progress tracking, but need further refinement for high-stakes clinical competence evaluation.
