FastAdaBelief: Improving Convergence Rate for Belief-Based Adaptive Optimizers by Exploiting Strong Convexity

Area of Science:

• Machine Learning
• Optimization Algorithms

Background:

• AdaBelief optimizer shows superior generalization to Adam by using exponential moving average of gradients.
• AdaBelief has a theoretical O(√T) regret bound for convex functions, but faster convergence remains an open problem.

Purpose of the Study:

• Introduce FastAdaBelief, a novel optimization algorithm designed to achieve faster convergence rates by exploiting strong convexity.
• Improve upon AdaBelief's convergence without compromising its generalization ability.

Main Methods:

• Developed FastAdaBelief with an adjusted step size considering strong convexity and preventing fluctuation.
• Theoretically proved a data-dependent O(logT) regret bound for FastAdaBelief in strongly convex cases.
• Empirically validated FastAdaBelief against mainstream algorithms using three baseline models in convex and non-convex scenarios.

Main Results:

• FastAdaBelief achieves a significantly lower O(logT) regret bound compared to AdaBelief's O(√T) in strongly convex settings.
• Experimental results show FastAdaBelief converges faster than all mainstream algorithms.
• FastAdaBelief maintains excellent generalization ability in both strong convexity and non-convexity.

Conclusions:

• FastAdaBelief is a novel optimization algorithm offering superior convergence and generalization.
• The theoretical O(logT) regret bound is a significant advancement for strongly convex problems.
• FastAdaBelief establishes a new benchmark for optimization research.