Determining a Clinically Applicable Cutoff in AI Algorithms for Predicting Clinical Deterioration: A...

Background: This retrospective study introduces an AI-driven VitalCare-Major Adverse Event Score (VC-MAES) developed to predict major in-hospital adverse events and determine optimal cutoff thresholds. VC-MAES was originally developed to predict a composite outcome including unplanned intensive care unit (ICU) transfer, in-hospital cardiac arrest, and mortality; the present study evaluates its performance for the composite of unplanned ICU transfer and in-hospital cardiac arrest. Methods: Patients aged ≥19 years in Yongin Severance Hospital between 1 March 2020, and 31 December 2022, were included. The primary outcome was clinica l deterioration, defined as unplanned ICU transfer or in-hospital cardiac arrest. Secondary outcomes included model performance metrics (area under the receiver operating characteristic (ROC) curve, sensitivity, specificity, F1 score) and optimal alarm frequency (number of alarms per 100 patient-days) across cutoff determination methods (Youden's index, F1 score, Euclidean distance, and alarm-based approach); they were used to determine optimal cutoff values. 20 July 2026. Results: VC-MAES achieved an area under the ROC curve of 0.895(full evaluable sample) at 6-h intervals, outperforming traditional early warning systems. It exhibited improved sensitivity (0.5853) compared to NEWS (0.3775) and MEWS (0.3049) while maintaining a specificity exceeding 0.95, balancing alarm frequency and predictive accuracy (all figures based on each model's own full evaluable sample: VC-MAES, n = 7425 events; NEWS, n = 3494; MEWS, n = 3198). On the common prediction time-point sample (n = 1678 events), the alarm-based cutoff yielded a sensitivity of 0.5244 for VC-MAES versus 0.2271 for NEWS and 0.1740 for MEWS, with alarm frequencies of 9.67, 7.07, and 5.58 per 100 patient-days, respectively. When compared on a common set of prediction time-points, VC-MAES (AUROC 0.832 on the common sample, 95% CI: 0.822-0.842) significantly outperformed NEWS (AUROC 0.743, 95% CI: 0.732-0.756; DeLong Z = -14.19, p < 0.0001) and MEWS (AUROC 0.706, 95% CI: 0.693-0.720; DeLong Z = -18.22, p < 0.0001). ROC-based methods, such as Youden's index and Euclidean distance, were deemed unsuitable for practical clinical use. Conclusions: Conventional ROC-based cutoff selection methods may not be suitable for real-world clinical implementation due to excessive alarm burden. An alarm-based cutoff approach provides a more clinically applicable strategy by balancing predictive performance with manageable alarm frequency. These findings highlight the importance of incorporating clinical usability into cutoff determination for AI-based early warning systems.