The Impact of Evaluation Strategy on Sepsis Prediction Model Performance Metrics in Intensive Care Data:...

Area of Science:

• Intensive care medicine
• Machine learning in healthcare
• Clinical informatics

Background:

• Sepsis prediction is a critical task in intensive care units (ICUs).
• Existing machine learning models for sepsis prediction use various evaluation strategies.
• There is no consensus on the optimal strategy for clinically meaningful performance assessment.

Purpose of the Study:

• To assess different evaluation approaches for sepsis prediction models.
• To compare model performance on a German ICU dataset (BerlinICU) versus a public dataset (MIMIC-IV).
• To examine the impact of evaluation strategies on performance metrics.

Main Methods:

• Retrospective, observational cohort study.
• Machine learning models pretrained on MIMIC-IV dataset applied to BerlinICU data.
• Comparison of fixed horizon, peak score, and continuous evaluation strategies.

Main Results:

• BerlinICU dataset: 40,132 admissions, 10.3% sepsis prevalence (Sepsis-3 definition).
• Temporal convolutional network model achieved AUROC of 0.67 on BerlinICU (continuous evaluation, 6-hour horizon) vs. 0.84 on MIMIC-IV.
• Fixed horizon evaluation showed reduced AUROC (0.61) on BerlinICU; peak score evaluation showed similar AUROC to continuous evaluation.
• Performance metrics improved with shorter prediction horizons.

Conclusions:

• Evaluation strategy significantly impacts sepsis prediction model performance metrics.
• Continuous evaluation aligns best with real-world clinical monitoring.
• Fixed horizon and peak score evaluations can produce skewed results, especially peak score with longer patient stays.