Penalized Regression Methods With Modified Cross-Validation and Bootstrap Tuning Produce Better Prediction Models

Area of Science:

• Statistics
• Biostatistics
• Machine Learning

Background:

• Maximum likelihood estimation (MLE) in risk prediction models often leads to overfitting, resulting in extreme predictions and poor calibration (calibration slope < 1).
• Penalized regression methods like Ridge and Lasso offer a solution by shrinking coefficients, but standard cross-validation tuning can over-shrink coefficients, causing high variability and poor calibration, especially with smaller sample sizes.

Purpose of the Study:

• To investigate if standard cross-validation's use of reduced training datasets causes over-estimation of the tuning parameter ($$) and excessive shrinkage in penalized regression models.
• To propose and evaluate a modified cross-validation tuning method that uses larger pseudo-development datasets to ensure cross-validation training datasets match the original development sample size.

Main Methods:

• A modified cross-validation tuning method was developed, estimating the tuning parameter ($$) from a bootstrap-derived pseudo-development dataset.
• This modified tuning ensures that the cross-validation training datasets are the same size as the original dataset, addressing potential over-estimation of $$.
• The performance of modified tuning and bootstrap tuning was compared against standard tuning for Ridge and Lasso regression in simulated and real datasets across various sample sizes.

Main Results:

• Modified and bootstrap tuning methods substantially improved the selection of the tuning parameter ($$) compared to standard tuning.
• These improved tuning methods resulted in a better calibration slope (CS) and reduced prediction errors compared to standard tuning.
• Both modified and bootstrap tuning demonstrated superior performance over maximum likelihood estimation (MLE) in terms of prediction accuracy.

Conclusions:

• The proposed modified cross-validation tuning method effectively addresses the over-shrinkage and calibration issues associated with penalized regression models.
• Modified and bootstrap tuning offer improved reliability and accuracy for risk prediction models, particularly in scenarios with limited or moderate sample sizes.
• These methods are easily implementable in standard statistical software, providing a practical enhancement for developing robust risk prediction models.