Adaptive Biased Stochastic Optimization.

Area of Science:

• Optimization Algorithms
• Machine Learning
• Convex Optimization

Background:

• Existing adaptive gradient methods (Adam, AdaGrad, RMSProp) form the basis for generalized adaptive gradient (GEAR) methods.
• Biased stochastic optimization (BSO) algorithms offer potential improvements in convergence properties.
• Analysis of these algorithms for composite objective functions is crucial for advancing optimization theory.

Purpose of the Study:

• To develop and analyze a novel class of adaptive biased stochastic optimization (ABSO) algorithms.
• To investigate the convergence rates and complexities of these ABSO algorithms for strongly convex (SC) and Polyak-Łojasiewicz (PŁ) composite objective functions.
• To introduce a new BSO algorithm, BSCG-GEAR, within the ABSO framework.

Main Methods:

• The study utilizes the GEneralized Adaptive gRadient (GEAR) framework to develop ABSO algorithms.
• Two specific BSO algorithms, BSVRG and SARAH, are integrated with GEAR, resulting in BSVRG-GEAR and SARAH-GEAR.
• A uniform theoretical analysis is performed for SC and PŁ composite objective functions.

Main Results:

• The developed ABSO algorithms, including BSVRG-GEAR and SARAH-GEAR, achieve linear convergence rates for both PŁ and SC composite objective functions.
• A novel algorithm, BSCG-GEAR, is introduced, matching established oracle complexity.
• The computational complexity of the proposed ABSO algorithms is shown to be competitive with state-of-the-art stochastic gradient methods.

Conclusions:

• The proposed ABSO algorithms demonstrate significant empirical superiority and numerical stability across various machine learning applications.
• The GEAR framework provides a unified approach for analyzing and developing advanced biased stochastic optimization algorithms.
• This work contributes to the theoretical understanding and practical application of efficient optimization techniques in machine learning.