Improving ICU Risk Predictive Models Through Automation Designed for Resiliency Against Documentation Bias

Area of Science:

• Critical care medicine
• Health informatics
• Biostatistics

Background:

• Electronic health records (EHRs) facilitate automated data capture for risk models but can introduce bias.
• Data drift and documentation changes (e.g., Glasgow Coma Scale) can impact the reliability of ICU benchmarking models.
• Existing models like APACHE may be susceptible to biases from EHR data variations.

Purpose of the Study:

• To present the Philips Critical Care Outcome Prediction Model (CCOPM), designed to mitigate bias from data drift in EHRs.
• To improve the accuracy and reliability of benchmarking Intensive Care Units (ICUs) on mortality performance.
• To evaluate the CCOPM's performance against established models like APACHE IV.

Main Methods:

• Retrospective, multicenter study using the eICU Research Institute database (509,586 adult ICU stays).
• Development of Generalized Additive Models (GAMs) using clinical features from the first 24 hours of ICU admission.
• Features were specifically designed to mitigate biases related to admission diagnosis, Glasgow Coma Scale (GCS), and extreme vital signs.

Main Results:

• CCOPM demonstrated superior discrimination for ICU mortality (AUROC 0.925 vs. 0.88) and hospital mortality (AUROC 0.90 vs. 0.86) compared to APACHE IVa/IVb.
• The model showed better calibration across diverse subgroups (admission diagnoses, ICU types) and over time (ICU-years).
• CCOPM provided more stable predictions than APACHE IVa in an external cohort with known GCS documentation shifts.

Conclusions:

• The CCOPM exhibits excellent performance and effectively mitigates bias from significant shifts in GCS documentation.
• These findings support the use of automated GCS data capture for reliable ICU mortality benchmarking.
• CCOPM offers a robust alternative for risk prediction in critical care settings, enhancing benchmarking accuracy.