A Small-Sample Choice of the Tuning Parameter in Ridge Regression

Area of Science:

• Statistics
• Machine Learning

Background:

• Ridge regression is a penalized likelihood method for regularizing linear regression coefficients.
• Existing methods for choosing the shrinkage parameter can be suboptimal in small-n, large-p settings, leading to insufficient or excessive shrinkage.
• This can negatively impact the predictive performance of the model.

Purpose of the Study:

• To propose novel approaches for selecting the shrinkage parameter in ridge regression, specifically for small sample sizes with a large number of parameters (small-n, large-p).
• To address the limitations of existing methods that may result in extreme shrinkage parameter choices.
• To enhance prediction accuracy in challenging small-sample regression scenarios.

Main Methods:

• A modified generalized cross-validation (GCV) method is proposed, correcting for small-n effects while preserving asymptotic optimality.
• A novel concept of 'hyperpenalty' is introduced to shrink the shrinkage parameter itself.
• A simple algorithm is developed for jointly estimating the shrinkage parameter and regression coefficients within the hyperpenalized likelihood framework.

Main Results:

• The proposed corrected GCV method maintains the desirable asymptotic properties of the original GCV.
• The introduced hyperpenalty, with a specific recommended choice, demonstrates empirical effectiveness across various scenarios.
• A comprehensive simulation study shows that the new approaches provide superior prediction performance compared to nine other existing methods.

Conclusions:

• The proposed methods offer improved shrinkage parameter selection for ridge regression in small-n, large-p situations.
• These approaches lead to enhanced predictive accuracy, particularly in small-sample scenarios.
• The hyperpenalty concept provides a robust way to regularize the regularization parameter itself, improving model stability and performance.