Calibration Drift Among Regression and Machine Learning Models for Hospital Mortality

Area of Science:

• Health Informatics
• Clinical Epidemiology
• Biostatistics

Background:

• Personalized risk prediction models are crucial for clinical decision-making.
• Model performance can degrade over time due to shifts in patient populations.
• Understanding calibration drift is essential for effective model updating.

Purpose of the Study:

• To evaluate how different regression and machine learning modeling methods affect calibration deterioration over time.
• To inform best practices for updating predictive models in healthcare.

Main Methods:

• Developed seven common regression and machine learning models for 30-day hospital mortality prediction.
• Models were trained on 2006 Department of Veterans Affairs hospital admissions and validated on 2007-2013 data.
• Assessed model discrimination and calibration stability.

Main Results:

• All models maintained discrimination over time.
• Calibration remained stable for the neural network model but declined for others.
• L-2 penalized logistic regression and random forest models showed less calibration drift than other regression models.
• Calibration drift was primarily associated with changes in patient case mix.

Conclusions:

• The choice of modeling method influences the stability of risk prediction model calibration.
• Neural networks offer superior calibration stability compared to other tested methods.
• Model updating strategies must account for method-specific calibration drift patterns.
• Changing patient case mix is a key driver of calibration drift.