Fully corrective gradient boosting with squared hinge: Fast learning rates and early stopping.

Area of Science:

• Machine Learning
• Computer Science

Background:

• Boosting methods are widely used for binary classification.
• Existing methods may face challenges with convergence rates and loss function properties.

Purpose of the Study:

• To propose an efficient boosting method for binary classification.
• To provide theoretical guarantees for the proposed method's performance.

Main Methods:

• Utilizing a fully corrective greedy (FCG) update for accelerated convergence.
• Employing a differentiable squared hinge loss function for robustness and theoretical benefits.
• Implementing an efficient alternating direction method of multipliers (ADMM) solver.

Main Results:

• The FCG update accelerates numerical convergence compared to traditional methods.
• The squared hinge loss combines the robustness of hinge loss with the benefits of square loss.
• Theoretical analysis proves a fast learning rate of order O((m/logm)^-1/2).
• Numerical experiments validate the method's outperformance on simulations and real data.

Conclusions:

• The proposed boosting method offers an efficient and theoretically sound approach for binary classification.
• The combination of FCG update, squared hinge loss, and ADMM solver leads to superior performance.