Comparing Single-Hospital and National Models to Predict 30-Day Inpatient Mortality

Advances in artificial intelligence and machine learning have facilitated the creation of mortality prediction models which are increasingly used to assess quality of care and inform clinical practice. One open question is whether a hospital should utilize a mortality model trained from a diverse nationwide dataset or use a model developed primarily from their local hospital data.

To compare performance of a single-hospital, 30-day all-cause mortality model against an established national benchmark on the task of mortality prediction.

We developed a single-hospital mortality prediction model using 9975 consecutive inpatient admissions at the Department of Veterans Affairs Palo Alto Healthcare System from July 26, 2018, to September 30, 2021, and compared performance against an established national model with similar features.

Both the single-hospital model and the national model placed each patient in one of five prediction bins: < 2.5%, 2.5-5%, 5-10%, 10-30%, and ≥ 30% risks of 30-day mortality. Evaluation metrics included receiver operator characteristic area under the curve (ROC AUC), sensitivity, specificity, and balanced accuracy. Final comparisons were made between the single-hospital model trained on the full training set and the national model for both metrics and prediction overlap.

With sufficiently large training sets of 2720 or greater inpatient admissions, there was no statistically significant difference between the performances of the national model (ROC AUC 0.89, 95%CI [0.858, 0.919]) and single-hospital model (ROC AUC 0.878, 95%CI [0.84, 0.912]). For the 89 mortality events in the test set, the single-hospital model agreed with the national model risk assessment or an adjacent risk assessment in 92.1% of the encounters.

A single-hospital inpatient mortality prediction model can achieve performance comparable to a national model when evaluated on a single-hospital population, given sufficient sample size.