Evaluating sepsis watch generalizability through multisite external validation of a sepsis machine learning model

Area of Science:

• Medical Informatics
• Clinical Decision Support Systems
• Machine Learning in Healthcare

Background:

• Sepsis is a leading cause of mortality and significant healthcare expenditure globally.
• Early detection of sepsis is critical for improving patient outcomes and reducing healthcare costs.
• Machine learning (ML) models offer potential for enhancing early sepsis detection.

Purpose of the Study:

• To validate the reproducibility and performance of Duke Health's Sepsis Watch ML model in a community healthcare setting.
• To assess the clinical utility of the Sepsis Watch model for early sepsis detection in emergency departments.
• To evaluate the model's portability across different geographical and demographic contexts.

Main Methods:

• Analysis of 205,005 patient encounters (2020-2021) from 101,584 unique patients at Summa Health.
• Utilized a combination of static and dynamic patient data for sepsis prediction.
• Evaluated model performance using Area Under the Receiver Operating Characteristic Curve (AUROC) and Area Under the Precision-Recall Curve (AUPRC).

Main Results:

• The Sepsis Watch ML model achieved high performance, with AUROC ranging from 0.906 to 0.960.
• AUPRC values ranged from 0.177 to 0.252 across four study sites.
• The model demonstrated strong and consistent performance, confirming its reproducibility in a new healthcare setting.

Conclusions:

• The Sepsis Watch ML model is reproducible and performs robustly in a community health system.
• The model's performance remained consistent across different settings, indicating its portability.
• The findings support the clinical utility of the Sepsis Watch model for early sepsis detection.