Using workload capacity indicators to evaluate rule-based early warning tools and their relationship to escalation...

Area of Science:

• Medical Informatics
• Clinical Decision Support
• Patient Safety

Background:

• Early Warning Tools (EWTs) are crucial for detecting patient deterioration.
• Evaluating EWTs traditionally focuses on accuracy metrics like AUROC and PPV.
• Hospital workload capacity is an important, yet often overlooked, factor in EWT performance.

Purpose of the Study:

• To compare the prediction accuracy of four rule-based EWTs using a large electronic medical record (EMR) dataset.
• To re-evaluate EWT performance using a novel hospital workload capacity evaluation method.
• To inform healthcare organizations on selecting appropriate EWTs and alert thresholds.

Main Methods:

• Retrospective analysis of adult inpatient admissions across 11 Australian hospitals.
• Evaluation of four EWTs: National Early Warning Score (NEWS), Between the Flags (BTF), Modified Early Warning Score (MEWS), and Queensland Adult Deterioration Detection Systems (Q-ADDS).
• EWTs assessed using Area Under the Receiver Operating Curve (AUROC), sensitivity, positive predictive value (PPV), and a novel clinician workload capacity analysis.

Main Results:

• NEWS2 demonstrated superior AUROC compared to Q-ADDS, MEWS, and BTF.
• Q-ADDS exhibited superior PPV at each alert threshold.
• Q-ADDS and MEWS operated at the lowest alert burden, indicating better workload capacity.

Conclusions:

• A precision-recall workload capacity analysis visually displays EWT operational characteristics.
• Healthcare organizations should consider clinician workload capacity alongside traditional accuracy metrics when selecting EWTs.
• This approach aids in setting effective EWT alert thresholds for improved patient care and resource management.