Fast binary logistic regression.

Area of Science:

• Machine Learning
• Statistics
• Numerical Analysis

Background:

• Binary logistic regression is a widely used statistical model in machine learning.
• Traditional training methods can be computationally intensive, especially for large datasets.
• Feature collinearity and model regularization are common challenges in logistic regression.

Purpose of the Study:

• To develop a novel numerical approach for significantly improving the training efficiency of binary logistic regression.
• To enable faster model parameter estimation and regularization through matrix-vector formulation.
• To address computational challenges associated with large datasets and collinear features.

Main Methods:

• Employed a novel Soft-Plus approximation to reformulate parameter estimation into matrix-vector form.
• Utilized the Lf-norm penalty for flexible regularization (L2, L1, L0 norms), including intercept penalization options.
• Applied Singular Value Decomposition (SVD), including randomized SVD and a new SVD with row reduction (SVD-RR), to handle collinearity and reduce complexity.
• Developed a Fast Binary Logistic Regression (FBLR) algorithm.

Main Results:

• Achieved training times an order of magnitude faster than traditional logistic regression.
• Demonstrated computational efficiency and effectiveness on diverse synthetic and OpenML datasets.
• Successfully managed datasets with numerous rows and features using SVD-RR.
• Provided a flexible framework for regularization and intercept handling.

Conclusions:

• The proposed FBLR method offers substantial improvements in training speed and computational efficiency.
• The novel numerical approach and regularization techniques provide a robust and flexible tool for binary logistic regression.
• The method is effective across various datasets, highlighting its generalizability and practical applicability in machine learning.