Prospective External Validation of an AI-Based Emergency Department Pneumonia Disposition Prediction Tool

Purpose: This shadow deployment evaluated an externally-developed AI tool to predict disposition using chest X-rays (CXR) in patients with community-acquired pneumonia (CAP) in the Emergency Department (ED). Retrospective and prospective external validations were conducted to assess differences between the 2 evaluations and across subgroups to inform deployment decisions. Methods: The CNN was retrospectively validated (n = 17 689) from November 1, 2020, to June 30, 2021, and prospectively validated on "suspected-CAP" patients (n = 3062) from Jan 1 to Jan 31, 2023. Calibration and standard metrics, including AUC, accuracy, sensitivity, specificity, PPV, and NPV, were calculated. Subgroup analyses were conducted for age, sex, modality, and CXR projection (PA vs AP). Results: The model's AUC was 67% in both validations. The prospective evaluation showed a non-significant increase in sensitivity (65% vs 59%) and PPV (64% vs 63%), while specificity (68% vs 73%) and NPV (69% vs 70%) slightly decreased. NPV was very high for younger patients in the prospective evaluation (95%); PPV was moderately high for older patients (81%). Sensitivity dropped significantly in females under 31 years (50%), and specificity was reduced in females over 86 years (38%). Conclusion: This study showed moderate, consistent performance in both retrospective and prospective validations. While this consistency is encouraging, further direct comparisons are needed to determine whether both validation approaches are necessary in different clinical settings. Subgroup analysis suggests the tool may be helpful to accelerate discharge in younger patients (high NPV) and possibly for admission in older patients (high PPV).