Using OR Black Box Technology to Determine Quality Improvement Outcomes for In-situ Timeout and Debrief Simulation

The purpose of this study was to determine quality improvement outcomes following the pilot implementation of an in-situ simulation designed to enhance surgical safety checklist performance.

OR Black Box (ORBB) technology allows near real-time assessment for surgical safety checklist performance. Before our study, timeout quality was 73.3%, compliance was 99.9%, and engagement was 89.7% (n=1993 cases); Debrief Quality was 76.0%, compliance was 66.9%, and engagement was 66.7% (n=1842 cases).

This IRB-approved study used prospective convergent multi-methods. During 2 months, a 15-minute in-situ simulation, incorporating rapid cycle deliberate practice, was implemented for OR teams. ORBB analytics generated Timeout and Debrief scores for actual operations performed by surgeons who participated in simulation (Sim-group) versus those who did not (No-sim group) over 6 months, including 2 months pre-intervention, during-intervention, and post-intervention. Inductive content analysis was performed based on simulation discussions to determine team member perspectives.

Thirty simulations with 163 interprofessional participants were conducted. ORBB data from 1570 cases were analyzed. Scores were significantly better for the Sim-group compared with the No-sim group for debrief quality (84% vs. 79% P <0.001, during-intervention), compliance (73% vs. 66%, P <0.001, post-intervention), and engagement (80% vs. 73%, P =0.012, during-intervention). There were no between-group differences for Timeout scores. Thematic analysis identified 2 primary categories: "culture of safety" and "policy."

This simulation-based QI intervention created a psychologically safe training environment for OR teams. The novel use of ORBB technology facilitated outcome analysis and showed significantly better Debrief scores for simulation-trained surgeons compared with nontrained surgeons.