Early warning score validation methodologies and performance metrics: a systematic review

Area of Science:

• Clinical prognostication
• Healthcare informatics
• Medical decision support

Background:

• Early warning scores (EWS) identify acutely deteriorating patients.
• Numerous studies develop and validate novel machine learning-based EWS.
• Conflicting conclusions in systematic reviews stem from varied validation methods.

Purpose of the Study:

• To examine methodologies and metrics used in EWS validation studies.
• To identify key differences in EWS validation approaches.
• To provide insights for standardizing EWS validation.

Main Methods:

• Systematic review of MEDLINE and other databases.
• Inclusion criteria: EWS validation studies reporting associations with mortality, ICU transfers, or cardiac arrest.
• Data abstraction using TRIPOD checklist; meta-analysis not performed due to heterogeneity.

Main Results:

• Key differences in validation: dataset, outcomes, case definition, EWS timing/aggregation, missing data handling.
• 48 studies reviewed; 34 used patient episode case definition, 12 used observation set.
• Over 10 different performance metrics reported, highlighting methodological diversity.

Conclusions:

• Heterogeneous validation methodologies and metrics hinder EWS performance interpretation and comparison.
• Standardization of EWS validation methodology and reporting is recommended.
• Standardization can improve the reliability and comparability of EWS performance.