Variable selection in linear regression models: Choosing the best subset is not always the best choice

Area of Science:

• Statistics
• Machine Learning
• Computational Statistics

Background:

• Variable selection is crucial for interpretable linear regression models.
• Best subset selection (BSS) is theoretically optimal but computationally expensive.
• Lasso and Elastic net are popular alternatives, especially in high-dimensional data.

Purpose of the Study:

• To conduct a neutral comparison of variable selection methods.
• To assess the performance of BSS against Forward Stepwise Selection (FSS), Lasso, and Elastic net (Enet).
• To evaluate methods under challenging conditions including high dimensionality, varying signal-to-noise ratios, and predictor correlations.

Main Methods:

• Simulations were used to compare BSS, FSS, Lasso, and Enet.
• Performance was primarily measured by the best possible F1-score.
• Alternative performance measures and practical criteria for parameter tuning were also used.

Main Results:

• BSS only reliably outperformed other methods in high signal-to-noise ratio settings with uncorrelated variables.
• Forward stepwise selection (FSS) performed nearly identically to BSS.
• Elastic net (Enet) showed advantages in speed and performance with correlated variables.

Conclusions:

• The presumption that BSS is always the best choice for variable selection is questioned.
• For correlated predictors, Enet is a faster and often better practical alternative.
• The study provides new insights into the comparative performance of common variable selection techniques.