Related Experiment Videos
Development and external validation of a multimodal artificial intelligence mortality prediction model of critically
Behrooz Mamandipoor1,2, Chun-Nan Hsu2, Martin Krause2,3
1Department of Biomedical Informatics, University of California, San Diego Health, La Jolla, CA, USA.
Background:
Early prediction of in-hospital mortality in critically ill patients can aid clinicians in optimizing treatment. The objective was to develop a multimodal deep learning model, using structured and unstructured clinical data, to predict in-hospital mortality risk among critically ill patients after their initial 24 hour intensive care unit (ICU) admission.
Methods:
We used data from MIMIC-III, MIMIC-IV, eICU, and HiRID. A multimodal model was developed on the MIMIC datasets, featuring time series components occurring within the first 24 hours of ICU admission and predicting risk of subsequent inpatient mortality. Inputs included time-invariant variables, time-variant variables, clinical notes, and chest X-ray images. External validation occurred in a temporally separated MIMIC population, HiRID, and eICU datasets. Area under the receiver operating characteristics (AUROC), area under the precision-recall curves (AUPRC), and Brier scores were reported.
Results:
A total of 203,434 ICU admissions from more than 200 hospitals between 2001 to 2022 were included, in which mortality rate ranged from 5.2% to 7.9% across the four datasets. The model integrating structured data points had AUROC, AUPRC, and Brier scores of 0.92 [95% confidence interval (CI) 0.90 - 0.93], 0.53 [95% CI 0.49 - 0.57], and 0.19 [95% CI 0.18 - 0.20], respectively. We externally validated the model on eight different institutions within the eICU dataset, demonstrating AUROCs ranging from 0.84-0.92. When including only patients with available clinical notes and imaging data, inclusion of notes and imaging into the model, the AUROC, AUPRC, and Brier score improved from 0.87 [95% CI 0.85 - 0.89] to 0.89 [95% CI 0.87 - 0.91], 0.43 [95% CI 0.36 - 0.51] to 0.48 [95% CI 0.40 - 0.56], and 0.37 [95% CI 0.36 - 0.39] to 0.17 [95% CI 0.16 - 0.19], respectively. The difference in AUROC was statistically significant based on DeLong test (p=0.0167).
Conclusions:
Our findings highlight the importance of incorporating multiple sources of patient information for mortality prediction and the importance of external validation.