External Validation of a Machine Learning Model for Delirium Risk

Although external validation of machine learning (ML)-based delirium risk models is a key stage in the model development pipeline, few published reports exist. Moreover, ML-based delirium risk models commonly degrade in performance during external validation.

To examine the performance of a fusion multimodal with natural language processing (NLP) delirium risk stratification model in live clinical practice with cross-hospital external validation.

This quality improvement study evaluated live clinical deployment of the model among patients 60 years or older who were admitted to non-intensive care units at Mount Sinai Hospital (MSH), with external validation performed at Mount Sinai Morningside Hospital (MSM), from December 1, 2024, to December 31, 2025.

Primary outcomes were performance of the risk stratification model, assessed by area under the receiver operating characteristic curve (AUROC), and monthly delirium detection rates. A 2 × 2 confusion matrix was created, and the 4 cells were used to calculate model performance metrics and compare associated diagnostic yield between premodel deployment and active model deployment periods.

The overall sample included 33 095 admissions to non-intensive care units at MSM and MSH (mean [SD] age, 74.86 (9.22) years; 15 601 women [47.1%], 17 271 men [52.2%]). A total of 9020 MSM inpatient admissions were assigned a delirium risk score by the risk stratification model during live clinical operations and were compared to 21 186 MSH inpatient admissions that were assigned a risk score by the same model. Age at admission significantly differed between patients at MSM and MSH (MSM: median [IQR] age, 75.79 [68.25-83.88] years vs MSH: 72.62 [66.64-79.65] years; P < .001). Patients from MSM and MSH also significantly differed by sex, race and ethnicity, surgical vs medical primary diagnosis, length of stay, and dementia prevalence rate. Among 11 909 MSM patients, patients in the pre-ML cohort (median [IQR] age, 78.00 [71.00-85.00] years) were significantly older than those in the post-ML cohort (median [IQR] age, 75.79 [68.25-83.88] years) (P < .001). The AUROC of the model was 0.91 (95% CI, 0.90-0.92) for MSM and 0.94 (95% CI, 0.93-0.94) for MSH. The positive predictive value of the model was 25.9% for MSM and 19.4% for MSH. For MSM, the mean monthly delirium detection rate significantly increased from 9.23% (95% CI, 8.07%-10.38%) to 19.59% (95% CI, 17.32%-21.85%) after model deployment (P < .001).

In this quality improvement study, an ML-based delirium risk stratification model maintained acceptable discriminative performance during external deployment without retraining and was similar to its performance at the development site without retraining. Model deployment was also associated with increased diagnostic yield but a high false positive rate. These findings provide further evidence of the potential generalizability of the model.