Sequential safe feature elimination rule for L₁-regularized regression with Kullback-Leibler divergence.

Area of Science:

• Machine Learning
• Statistical Modeling
• Computational Statistics

Background:

• L1-regularized regression with Kullback-Leibler divergence (KL-L1R) is widely used.
• Efficient implementation of KL-L1R is challenging with a high number of features.
• Existing methods struggle with scalability for extremely large feature sets.

Purpose of the Study:

• To develop a novel and fast sequential safe feature elimination rule (FER) for accelerating KL-L1R.
• To significantly reduce computational time for KL-L1R models with numerous features.
• To ensure the proposed method maintains the exact solution of the original KL-L1R.

Main Methods:

• Introduction of a sequential safe feature elimination rule (FER) leveraging sparsity and duality theory.
• Efficient selection and removal of redundant features before and during model training.
• Utilizing the Newton coordinate descent method (Newton-CDM) to solve the reduced model.

Main Results:

• FER significantly reduces computational time by eliminating most redundant features.
• The proposed method requires solving only one reduced model.
• Numerical experiments confirm the feasibility and validity of FER across diverse datasets.

Conclusions:

• The novel FER provides a safe and efficient way to accelerate KL-L1R.
• This approach effectively handles datasets with extremely large numbers of features.
• FER offers a practical solution for improving the performance of KL-L1R in high-dimensional settings.