Penalization and shrinkage methods produced unreliable clinical prediction models especially when sample size was...

Area of Science:

• Clinical epidemiology
• Statistical modeling
• Biostatistics

Background:

• Penalization techniques are recommended for clinical prediction models to prevent overfitting.
• These methods shrink effect estimates and reduce prediction error.
• However, tuning parameters in penalization methods are estimated with uncertainty.

Purpose of the Study:

• To examine the uncertainty in tuning parameter estimation for penalization methods.
• To assess the impact of this uncertainty on clinical prediction model performance.

Main Methods:

• Applied examples and simulation study.
• Evaluated uniform shrinkage, ridge regression, lasso, and elastic net.
• Investigated the influence of effective sample size and model performance metrics.

Main Results:

• Tuning parameter estimation in penalization methods can be unreliable due to uncertainty.
• This unreliability is most pronounced in small effective sample sizes and low model performance (low Cox-Snell R²).
• Uncertainty can lead to significant miscalibration of predictions in new individuals.

Conclusions:

• Penalization methods do not guarantee reliable prediction models.
• Their reliability decreases when overfitting is most likely (small sample sizes).
• Recommend using penalization methods with large effective sample sizes to minimize overfitting and ensure precise parameter estimation.