A Forward and Backward Stagewise Algorithm for Nonconvex Loss Functions with Adaptive Lasso

Area of Science:

• Statistics
• Machine Learning
• Computational Statistics

Background:

• Penalization methods are widely used for analyzing multi- and high-dimensional data.
• Existing algorithms primarily focus on convex loss functions, limiting applications for nonconvex scenarios.
• Nonconvex loss functions offer potential for more robust results and have significant practical applications.

Purpose of the Study:

• To develop a novel computational algorithm, Forward and Backward Stagewise (Fabs), for nonconvex loss functions coupled with the adaptive Lasso (aLasso) penalty.
• To analyze the theoretical properties of the Fabs algorithm, including its convergence and approximation guarantees.
• To demonstrate the practical utility of Fabs through applications in penalized smooth partial rank (SPR) estimation and binary classification.

Main Methods:

• Development of the Forward and Backward Stagewise (Fabs) algorithm, inspired by the BLasso algorithm.
• Theoretical analysis establishing that Fabs paths yield δ-approximate solutions to the aLasso problem.
• Convergence analysis showing Fabs paths converge to stationary points as δ approaches zero under bounded second-order derivatives.
• Application and numerical studies using Fabs for penalized smooth partial rank (SPR) estimation and smoothed 0-1 loss in binary classification.

Main Results:

• The Fabs algorithm provides a computational framework for nonconvex penalized regression.
• Each path point generated by Fabs is a δ-approximate solution to the aLasso problem.
• Fabs demonstrates effectiveness in penalized SPR estimation, especially in high-dimensional settings.
• The algorithm's capability to handle other differentiable nonconvex loss functions, such as the smoothed 0-1 loss, is confirmed.

Conclusions:

• The Fabs algorithm offers an effective computational solution for high-dimensional data problems involving nonconvex loss functions and aLasso penalties.
• Fabs shows particular promise for penalized SPR estimation and binary classification tasks.
• The study highlights the broader applicability of Fabs to various differentiable nonconvex loss functions in statistical modeling.