Related Experiment Video
Updated: Aug 28, 2025
Implementation of a Real-Time Psychosis Risk Detection and Alerting System Based on Electronic Health Records using CogStack
Published on: May 15, 2020
Development and validation of an early warning model for hospitalized COVID-19 patients: a multi-center retrospective
Jim M Smit1,2, Jesse H Krijthe3, Andrei N Tintu4
1Department of Intensive Care, Erasmus University Medical Center, Rotterdam, The Netherlands. j.smit@erasmusmc.nl.
Insights
A new early warning model for COVID-19 patients demonstrated superior performance compared to general scores. This specialized model aids in timely identification of deteriorating patients, improving clinical management and ICU admission decisions.
Area of Science:
- Medical Informatics
- Clinical Prediction Models
- Epidemiology
Background:
- Timely identification of deteriorating COVID-19 patients is crucial for effective clinical management and intensive care unit (ICU) admission.
- Existing Early Warning Scores (EWSs) may underestimate illness severity in COVID-19 patients.
- Development of a specific early warning model for COVID-19 patients is warranted.
Purpose of the Study:
- To develop and validate a novel early warning model tailored for COVID-19 patients.
- To compare the performance of the new model against established EWSs.
- To investigate methods for dynamic model updating to maintain performance over time.
Main Methods:
- Retrospective collection of electronic medical record data from 3514 COVID-19 admissions across six Dutch hospitals.
- Development of a random forest model using 18 predictors.
- Temporal validation simulating real-world implementation during different COVID-19 waves, employing dynamic updating strategies.
Main Results:
- The COVID-19 specific model achieved a higher discriminative performance (partial AUC 0.82) than the National early warning score (0.72) and Modified early warning score (0.67).
- The model demonstrated greater net benefit across clinically relevant thresholds and good calibration.
- SHapley Additive exPlanations values were used to quantify predictor importance.
Conclusions:
- Specialized early warning models for specific patient groups, like COVID-19, offer potential benefits over general inpatient models.
- The developed model shows promise for improving patient monitoring and care.
- Further independent validation is recommended to confirm the model's generalizability and clinical utility.
Background:
Timely identification of deteriorating COVID-19 patients is needed to guide changes in clinical management and admission to intensive care units (ICUs). There is significant concern that widely used Early warning scores (EWSs) underestimate illness severity in COVID-19 patients and therefore, we developed an early warning model specifically for COVID-19 patients.
Methods:
We retrospectively collected electronic medical record data to extract predictors and used these to fit a random forest model. To simulate the situation in which the model would have been developed after the first and implemented during the second COVID-19 'wave' in the Netherlands, we performed a temporal validation by splitting all included patients into groups admitted before and after August 1, 2020. Furthermore, we propose a method for dynamic model updating to retain model performance over time. We evaluated model discrimination and calibration, performed a decision curve analysis, and quantified the importance of predictors using SHapley Additive exPlanations values.
Results:
We included 3514 COVID-19 patient admissions from six Dutch hospitals between February 2020 and May 2021, and included a total of 18 predictors for model fitting. The model showed a higher discriminative performance in terms of partial area under the receiver operating characteristic curve (0.82 [0.80-0.84]) compared to the National early warning score (0.72 [0.69-0.74]) and the Modified early warning score (0.67 [0.65-0.69]), a greater net benefit over a range of clinically relevant model thresholds, and relatively good calibration (intercept = 0.03 [- 0.09 to 0.14], slope = 0.79 [0.73-0.86]).
Conclusions:
This study shows the potential benefit of moving from early warning models for the general inpatient population to models for specific patient groups. Further (independent) validation of the model is needed.
More Related Videos
Related Concept Videos
01:18Steps in Outbreak Investigation
01:03Data Validation
Nursing assessment guides are generally based on holistic models rather than medical...