High-dimensional regression in practice: an empirical study of finite-sample prediction, variable selection and...

Area of Science:

• Statistics
• Machine Learning
• Computational Biology

Background:

• Penalized likelihood methods are crucial for high-dimensional regression analysis.
• Existing theory is well-developed, but practical performance in finite samples is unclear.
• Empirical studies are needed to guide users in selecting appropriate methods.

Purpose of the Study:

• To conduct a large-scale empirical comparison of various penalized regression methods.
• To evaluate method performance for prediction, variable selection, and variable ranking.
• To provide practical insights into method suitability based on data characteristics.

Main Methods:

• Comparison of Lasso, Adaptive Lasso, Elastic Net, Ridge Regression, SCAD, Dantzig Selector, and Stability Selection.
• Utilized over 2300 synthetic and semi-synthetic data-generating scenarios.
• Systematically varied factors including sample size, dimensionality, sparsity, signal strength, and multicollinearity.

Main Results:

• Significant performance variations observed among different penalized regression methods.
• No single method demonstrated superior performance across all scenarios and goals.
• Results indicate that method choice depends on the specific objective and data properties.

Conclusions:

• A 'no panacea' approach is supported; method selection requires careful consideration of the task and data.
• The study provides empirical evidence to complement theoretical understanding.
• Offers practical recommendations for choosing penalized regression techniques in high-dimensional settings.