Determining a Clinically Applicable Cutoff in AI Algorithms for Predicting Clinical Deterioration: A...

Area of Science:

• Artificial Intelligence in Medicine
• Clinical Decision Support Systems
• Patient Monitoring

Background:

• Development of the AI-driven VitalCare-Major Adverse Event Score (VC-MAES) for predicting major in-hospital adverse events.
• Evaluation of VC-MAES performance for the composite outcome of unplanned intensive care unit (ICU) transfer and in-hospital cardiac arrest.

Purpose of the Study:

• To assess the predictive performance of VC-MAES for clinical deterioration.
• To determine optimal cutoff thresholds for VC-MAES using various methods.
• To compare VC-MAES with traditional early warning systems (NEWS, MEWS).

Main Methods:

• Retrospective study of adult patients (≥19 years) from March 2020 to December 2022.
• Primary outcome: clinical deterioration (unplanned ICU transfer or in-hospital cardiac arrest).
• Analysis of model performance metrics (AUROC, sensitivity, specificity, F1 score) and alarm frequencies using different cutoff determination methods (Youden's index, F1 score, Euclidean distance, alarm-based approach).

Main Results:

• VC-MAES achieved an AUROC of 0.895, outperforming NEWS (AUROC 0.743) and MEWS (AUROC 0.706) on a common sample.
• VC-MAES demonstrated higher sensitivity (0.5244) compared to NEWS (0.2271) and MEWS (0.1740) with an alarm-based cutoff.
• Conventional ROC-based cutoff methods were found unsuitable for clinical practice due to excessive alarms; the alarm-based approach offered a better balance.

Conclusions:

• Conventional methods for selecting cutoff values for AI-based early warning systems may lead to excessive alarms in clinical settings.
• An alarm-based cutoff strategy is more clinically applicable, effectively balancing predictive performance with manageable alarm frequency.
• The study emphasizes the importance of considering clinical usability when determining cutoff values for AI-driven predictive models.