Post-Estimation Shrinkage in Full and Selected Linear Regression Models in Low-Dimensional Data Revisited.

Area of Science:

• Statistics
• Machine Learning

Background:

• Overfitting degrades regression model performance on new data.
• Variable selection can introduce bias into regression estimates.
• Shrinkage methods mitigate overfitting and bias.

Purpose of the Study:

• Evaluate post-estimation shrinkage for improving prediction performance.
• Compare shrinkage methods against ordinary least squares (OLS), ridge, best subset selection (BSS), and lasso.
• Introduce and assess a novel non-negative parameter-wise shrinkage (NPWS) method.

Main Methods:

• Simulation study comparing prediction errors and variable selection.
• Evaluation of full models with OLS, ridge, and shrinkage methods.
• Assessment of selected models with BSS, lasso, and post-estimation shrinkage.

Main Results:

• NPWS outperformed global shrinkage in full models; PWS was inferior to OLS.
• NPWS excelled over ridge in low correlation/high SNR; ridge was best in small samples/high correlation/low SNR.
• In selected models, post-estimation shrinkage methods performed similarly, with global shrinkage slightly inferior.
• Lasso outperformed BSS and post-estimation shrinkage in specific conditions (small samples, low SNR, high correlation).

Conclusions:

• NPWS enhances prediction accuracy more than global shrinkage when sufficient data is available.
• Penalized methods generally outperform post-estimation shrinkage in high correlation, small sample sizes, and low SNR scenarios.