Bayes Consistency vs. -Consistency: The Interplay between Surrogate Loss Functions and the Scoring Function Class.

Area of Science:

• Machine Learning
• Statistical Learning Theory

Background:

• Surrogate risk minimization is key for multiclass classification, aiming to approximate the 0-1 loss.
• Calibrated surrogates ensure Bayes consistency but may falter in achieving $\epsilon$-consistency, especially with linear models and linear scoring functions.
• Prior work highlighted limitations of certain calibrated surrogates in recovering underlying linear models.

Purpose of the Study:

• Investigate the limitations of calibrated surrogates in achieving $\epsilon$-consistency for multiclass classification.
• Propose a novel approach using carefully chosen scoring function classes to enable $\epsilon$-consistency.
• Enhance the performance of popular surrogates like one-vs-all hinge and logistic loss.

Main Methods:

• Introduced a specialized class of piecewise linear scoring functions for minimizing calibrated surrogates.
• Adapted min-pooling techniques from neural network training for optimization.
• Empirically evaluated the proposed method against standard linear scoring functions.

Main Results:

• Demonstrated that specific nonlinear scoring function classes enable $\epsilon$-consistency for calibrated surrogates.
• Showcased improved performance of one-vs-all hinge and logistic surrogates when trained on the derived nonlinear scoring functions.
• Achieved better linear multiclass classifiers compared to training with standard linear scoring functions.

Conclusions:

• The choice of scoring function class is critical for achieving $\epsilon$-consistency in multiclass classification.
• Nonlinear scoring functions, specifically piecewise linear ones, can overcome limitations of linear models with calibrated surrogates.
• The proposed method offers a practical way to improve multiclass classification accuracy using established surrogate losses.