Development and Validation of a Quantitative Grading Rubric for High-Fidelity Simulation Assessment

High-fidelity simulation scenarios provide trainees the opportunity to demonstrate foundational anesthesia skills, complex prioritization, and decision making. One strategy used in trainee assessment is an objective structured clinical examination (OSCE). The purpose of this study was to design an OSCE with a reliable, quantitative grading rubric that could be used as part of a comprehensive assessment strategy to determine readiness for entry into clinical training for nurse anesthesia residents. An observational correlational study was developed to validate four high-fidelity simulation scenarios and accompanying quantitative grading rubrics. The rubrics were tested on junior nurse anesthesia residents and graded by program faculty members. Interrater reliability was tested using Krippendorff's alpha. Three cohorts of nurse anesthesia residents (n = 83) were assessed yielding 330 unique observations. Interrater reliability increased over the duration of the study with an overall reliability coefficient of 0.9092 (95% CI, 0.8509-0.9062), indicating a very high degree of interrater reliability among a variety of raters in complex simulated environments. Development of a quantitative rubric for high-fidelity simulation is achievable and should be considered as part of a summative assessment to differentiate individual student performance and readiness to proceed to clinical training within a front-loaded nurse anesthesia program.