Complementary Error Patterns Between Human Evaluators and GPT-4o in Video-Based Cardiopulmonary Resuscitation Skills...

Background/Objectives: Cardiopulmonary resuscitation (CPR) skill assessments are susceptible to evaluator subjectivity, cognitive fatigue, and observational limitations. Although recent advances in multimodal artificial intelligence have increased the possibility of automated video-based assessment, its validity for clinical skill evaluation remains insufficiently examined. Methods: In this cross-sectional study, we enrolled 130 laypersons who underwent Basic Life Support training and skill testing. Twenty recordings were used for prompt development and 110 recordings were analyzed. Expert evaluators and GPT-4o independently assessed participants' skills using a 12-item checklist. The manikin sensor data were the reference standard for the four chest compression metrics. Agreement was evaluated using Gwet's agreement coefficient 1 (AC1) and intraclass correlation coefficient (2,1). Diagnostic accuracy, sensitivity, and specificity were compared using McNemar's test. Results: Procedural items such as confirming cardiac arrest, calling 119, and requesting an automated external defibrillator showed a near-perfect agreement between experts and GPT-4o (AC1 > 0.8). However, the agreement was poor for the compression depth (AC1 = 0.374) and full chest recoil (AC1 = 0.355). Experts demonstrated high sensitivity (77.8-84.3%) but low specificity (24.6-47.8%), whereas GPT-4o showed low sensitivity (35.6-40.6%) but high specificity (69.2-76.1%). Conclusions: GPT-4o cannot serve as a standalone evaluator because of its inherent limitations in inferring three-dimensional spatial information from two-dimensional videos. However, its high agreement on procedural items and complementary error patterns with that of human evaluators on compression metrics suggests its potential as a decision support tool to mitigate expert leniency bias in CPR education.