Comparing Machine Learning to Regression Methods for Mortality Prediction Using Veterans Affairs Electronic Health...

Area of Science:

• Health Informatics
• Biostatistics
• Machine Learning in Healthcare

Background:

• The comparative accuracy of machine learning (ML) versus traditional regression models for predicting outcomes using electronic health records (EHRs) is not well-established.
• This study addresses the uncertainty surrounding the performance of ML methods in EHR-based prediction models.

Purpose of the Study:

• To compare the predictive performance of ML models against traditional regression models for 10-year all-cause mortality using Veterans Affairs (VA) EHR data.
• To evaluate discrimination and calibration of various models.

Main Methods:

• A cohort study design was employed using VA EHR data from veterans aged over 50.
• Models were developed using a training cohort (n=124,360) and tested on a separate cohort (n=124,360).
• 924 potential predictors were analyzed, including demographics, vital signs, medications, diagnoses, lab results, and healthcare utilization. Model performance was assessed using c-statistics, calibration, and diagnostic characteristics.

Main Results:

• Gradient Boosting (ML) achieved the highest c-statistic (0.837) using all 924 predictors.
• The full logistic regression model had a c-statistic of 0.833, but exhibited overfitting.
• Stepwise logistic regression (101 predictors) demonstrated similar discrimination (0.832) with minimal overfitting, and all models showed good calibration and comparable diagnostic test characteristics.

Conclusions:

• The modest differences in predictive accuracy between the best ML model and stepwise logistic regression suggest that traditional methods remain effective for EHR mortality prediction.
• Further validation in non-VA EHR systems is recommended.