Longitudinal Model Shifts of Machine Learning-Based Clinical Risk Prediction Models: Evaluation Study of Multiple Use...

Area of Science:

• Clinical informatics
• Machine learning in healthcare
• Predictive modeling

Background:

• Machine learning (ML) models are crucial for predicting clinical risk events.
• Model performance degrades due to dynamic system and data changes in production.
• Monitoring model shifts is essential for maintaining clinical efficacy.

Purpose of the Study:

• Assess the impact of model shifts on ML prediction models.
• Evaluate delirium, sepsis, and acute kidney injury (AKI) models across two hospitals.
• Investigate model performance during the COVID-19 pandemic.

Main Methods:

• Trained models on retrospective data, tested on recent data.
• Evaluated model performance using Area Under the Receiver Operating Characteristic Curve (AUROC).
• Analyzed calibration curves, alert rates, diagnostic accuracy, and decision curves.

Main Results:

• AUROCs showed no significant performance differences across years for all use cases and hospitals.
• Calibration shifts were detected for delirium and sepsis models, but not AKI.
• Decision curve analysis (DCA) revealed no impact on clinical utility.

Conclusions:

• ML risk prediction models demonstrated stable performance (AUROC) despite evolving clinical practices.
• Calibration shifts in specific use cases did not compromise clinical decision-making.
• Continuous monitoring of data changes and model shifts is vital for clinical practice.