Massive external validation of a machine learning algorithm to predict pulmonary embolism in hospitalized patients

Area of Science:

• Medical Informatics
• Clinical Decision Support
• Predictive Analytics in Healthcare

Background:

• Pulmonary embolism (PE) is a leading cause of death in hospitalized patients.
• Early prediction of PE is crucial for timely intervention and improved outcomes.
• Generalizability of predictive models across diverse patient populations is key for clinical utility.

Purpose of the Study:

• To perform the first large-scale external validation of a machine learning-based PE prediction model.
• To assess the model's performance using EHR data from the initial hours of hospitalization.
• To determine the model's predictive capability for PE occurrence within 10 days of inpatient stay.

Main Methods:

• Retrospective analysis of approximately two million adult hospital admissions across 44 US institutions (2011-2017).
• Training an XGBoost model using demographics, vital signs, and lab tests from 331,268 patients at 12 institutions.
• External validation on 1,660,715 patients from 32 institutions without model retraining, assessing performance via AUROC.

Main Results:

• The model achieved an AUROC of 0.87 on the training set hold-out.
• Excellent discrimination was observed across 32 external validation sets, with a mean AUROC of 0.88 (range: 0.79-0.93).
• Mean specificity was 0.85 at 0.80 sensitivity, with backward elimination regression identifying a negative association between PE prevalence and AUROC.

Conclusions:

• The PE prediction model demonstrates remarkable performance and generalizability across diverse patient populations without retraining.
• The model's robustness suggests its potential as a valuable clinical decision support tool for PE detection.
• This validates the model's utility in improving patient outcomes within clinical settings.