Multimodal artificial intelligence assessment of simulated cardiac arrest performance: a pilot study

Assessment of simulated cardiac arrest performance is resource-intensive and subject to inter-rater variability. We evaluated whether a multimodal artificial intelligence (AI) system could score video-recorded cardiac arrest megacode simulations with agreement comparable to expert raters.

Six scripted megacode simulations (three ventricular fibrillation, three pulseless electrical activity) representing exemplar, guideline-deviation and non-technical-deficiency performance were scored by two ACLS instructors using a validated team-leader checklist; consensus ratings served as the reference. Two operators analysed each video with Gemini 1.5 Pro at three temperature settings (504 item-level ratings). We calculated Cohen's kappa, Gwet's AC1, and sensitivity and specificity for identifying performed and omitted actions, and compared AI self-reported confidence between correct and incorrect ratings.

Expert inter-rater agreement was κ = 0.78 (95% CI 0.64-0.93). AI-expert agreement was κ = 0.62 (95% CI 0.49-0.75; range 0.40-0.86 across configurations) and AC1 = 0.77. The AI identified performed actions with high sensitivity (0.98, 95% CI 0.96-0.99) but detected omitted actions poorly (specificity 0.58, 95% CI 0.44-0.72), particularly omissions defined by timing or sequence. Temperature had no consistent effect. Incorrect ratings of actions as performed carried lower mean self-reported confidence than correct ones (93.6 vs 98.1), but 48% were made with maximal confidence.

A multimodal AI system reliably confirmed completed actions but frequently failed to detect omitted actions, yielding moderate overall agreement that was below expert-expert agreement. Current systems may support, but not replace, expert assessment of cardiac arrest simulation.