Advancing neural network calibration: The role of gradient decay in large-margin Softmax optimization

Area of Science:

• Machine Learning
• Deep Learning
• Neural Network Optimization

Background:

• Modern neural networks often exhibit overconfidence and calibration issues.
• Large margin Softmax methods aim to improve discriminative power.
• Understanding gradient dynamics is crucial for model performance.

Purpose of the Study:

• Introduce a novel hyperparameter to control probability-dependent gradient decay in Softmax.
• Investigate the impact of gradient decay rate on model generalization and calibration.
• Explore the relationship between gradient decay, curriculum learning, and Lipschitz constraints.

Main Methods:

• Theoretical and empirical analysis of a new Softmax hyperparameter.
• Examining gradient decay behavior with varying sample probabilities (convex/concave).
• Proposing and evaluating a novel dynamic gradient decay warm-up strategy.

Main Results:

• Smaller gradient decay induces curriculum learning but exacerbates overconfidence.
• Larger gradient decay significantly improves model calibration, surpassing post-calibration techniques.
• Probability-dependent gradient decay influences the local Lipschitz constraint.

Conclusions:

• Gradient decay rate is a critical factor for both generalization and calibration in Softmax.
• Large gradient decay offers a promising approach to mitigate overconfidence in neural networks.
• The proposed warm-up strategy enhances training stability and final model calibration.