Predictive performance of machine and statistical learning methods: Impact of data-generating processes on external...

Area of Science:

• Cardiovascular medicine
• Machine learning
• Statistical modeling

Background:

• Machine learning methods are increasingly proposed for enhancing clinical outcome prediction.
• The comparative performance of various machine learning and statistical methods under different data-generating conditions remains an area of investigation.

Purpose of the Study:

• To determine when machine learning methods demonstrate superior predictive accuracy compared to classical learning methods.
• To examine the influence of data-generating processes on the relative performance of six distinct learning algorithms.

Main Methods:

• Simulations were conducted using two large cardiovascular datasets: acute myocardial infarction (AMI) and congestive heart failure (CHF).
• Six learning methods (bagged trees, gradient boosting machines, random forests, lasso, ridge, and logistic regression) were evaluated.
• Performance was assessed using c-statistic, generalized R-squared, Brier score, and calibration across simulated derivation and validation samples.

Main Results:

• No single method consistently outperformed all others across all data-generating processes and performance metrics.
• Unpenalized and penalized logistic regression, along with boosted trees, generally showed superior performance.
• Classical statistical learning methods demonstrated strong performance in low-dimensional settings with substantial data.

Conclusions:

• Classical statistical learning methods remain highly effective for clinical outcome prediction, particularly in large, low-dimensional datasets.
• The choice of prediction method should consider the underlying data-generating process and specific performance metrics.
• Boosted trees and logistic regression variants are robust choices for predicting cardiovascular outcomes.