Adaptive regularization using the entire solution surface.

Area of Science:

• Statistics
• Machine Learning
• Computational Statistics

Background:

• Automatic variable selection in high-dimensional data often relies on regularization techniques.
• Existing sparseness penalties typically assume a sparse underlying model, limiting their applicability.
• There is a need for regularization methods that adapt to varying degrees of sparsity and feature grouping.

Purpose of the Study:

• To propose a novel regularization penalty that combines L1 and L-infinity norms.
• To develop an efficient homotopy algorithm for computing regularization solution paths with multiple regularizers.
• To evaluate the performance of the proposed penalty against established methods in variable selection.

Main Methods:

• Introduction of a convex combination penalty using L1 and L-infinity norms.
• Development of a subgradient-based homotopy algorithm for efficient computation of regularization paths.
• Numerical experiments using simulated data and real-world examples for performance evaluation.

Main Results:

• The proposed penalty demonstrates adaptive regularization capabilities, handling both sparse and non-sparse scenarios.
• The method effectively performs feature grouping, a desirable property in many applications.
• The novel homotopy algorithm allows for efficient computation and adaptive tuning of the regularization parameters.

Conclusions:

• The proposed L1/L-infinity norm combination penalty offers a flexible and effective approach to variable selection.
• The adaptive nature of the penalty makes it suitable for a wider range of data structures compared to existing methods.
• The developed algorithm provides an efficient computational solution for regularization problems with multiple regularizers.