Are observation selection methods important when comparing early warning score performance?

Area of Science:

• Clinical assessment
• Patient monitoring
• Health informatics

Background:

• Sicker patients undergo more frequent vital sign assessments, especially before adverse events.
• Early warning scores (EWSs) often drive monitoring frequency, potentially biasing performance evaluations.
• The independence of vital sign observations is crucial for accurate EWS discrimination analysis.

Purpose of the Study:

• To evaluate the impact of observation selection methods on the discriminatory performance of 35 published early warning scores (EWSs).
• To determine if vital sign observations can be treated as independent when assessing EWS performance.
• To identify the most effective EWS for predicting in-hospital mortality.

Main Methods:

• Utilized a large dataset of 1,564,143 vital sign observations from routine patient care.
• Compared 35 EWSs using three observation selection strategies: all observations, one random observation per episode, and one time-closest observation per episode.
• Assessed discrimination of 24-hour mortality risk using the area under the ROC curve (AUROC) and analyzed rank order changes.

Main Results:

• No significant changes in EWS rank order were observed across selection methods, except for one score incorporating age.
• The National Early Warning Score (NEWS) consistently exhibited the highest discrimination for 24-hour mortality risk.
• AUROCs were generally higher when using a single observation set per patient episode, particularly for NEWS.

Conclusions:

• Vital sign measurements can be considered independent for comparing EWS performance and ranking.
• Multiple observations per patient episode are valid for EWS evaluation, provided the score does not include age.
• NEWS remains the most effective EWS for predicting short-term mortality risk.