Operational Integration and Temporal Validation of a Continuously Deployed ICU Prediction Model

Area of Science:

• Critical Care Medicine
• Health Informatics
• Machine Learning in Healthcare

Background:

• Intensive Care Units (ICUs) require continuous patient monitoring and accurate prognostication.
• Electronic Medical Records (EMRs) offer vast data for developing predictive models.
• Real-time outcome prediction can enhance clinical decision-making in acute illness.

Purpose of the Study:

• To operationalize and validate the BEST-AI (Big data-driven Evaluation of Survival and Treatment in Acute Illness) system.
• To assess the hourly prediction accuracy (discrimination and calibration) of the EMR-integrated machine learning system for multiple ICU outcomes.
• To evaluate the feasibility of integrating real-time predictions into the clinical workflow.

Main Methods:

• A single-center hybrid study involving stepwise clinical deployment and forward-in-time temporal validation.
• Development and validation cohorts from a tertiary mixed medical-surgical ICU (n=11,176 and n=1,127).
• EMR-integrated deployment of BEST-AI providing hourly probabilistic predictions without mandated interventions.

Main Results:

• Six prediction tasks evaluated, including mortality, intubation, and tracheostomy.
• Temporal validation showed strong discrimination (AUROC 0.856-0.960) and generally good calibration.
• The system was successfully maintained with automated hourly updates and EMR-embedded visualizations.

Conclusions:

• A continuously deployed, EMR-integrated ICU prediction system achieved strong temporal discrimination and good calibration.
• Embedding real-time predictions into routine ICU workflow is feasible.
• Prospective multicenter studies are needed to assess transportability and clinical impact.