Convergence of Adaptive Stochastic Mirror Descent

Area of Science:

• Optimization Theory
• Machine Learning Algorithms
• Convex Analysis

Background:

• Adaptive stochastic optimization methods are crucial for machine learning.
• Mirror maps capture geometric properties in optimization.
• Adaptive moment estimation (Adam)-type algorithms are widely used but lack theoretical grounding for hyperparameter choices.

Purpose of the Study:

• To present a family of adaptive stochastic optimization methods based on mirror maps.
• To theoretically analyze the convergence rates of these methods for various function types.
• To provide explanations for hyperparameter selection in Adam-type algorithms.

Main Methods:

• Developing adaptive stochastic optimization algorithms incorporating mirror maps.
• Analyzing average regret convergence rates for convex and strongly convex objective functions.
• Investigating convergence for smooth, non-convex functions using properties of strongly convex differentiable mirror maps.

Main Results:

• Achieved convergence rates for convex objective functions under standard assumptions.
• Improved convergence rates for strongly convex objective functions.
• Demonstrated convergence rates of order up to a logarithmic term for smooth objective functions, aligning with practical Adam-type algorithm usage.

Conclusions:

• The proposed family of adaptive stochastic optimization methods offers theoretical guarantees.
• The study provides insights into the effectiveness and hyperparameter choices of Adam-type algorithms.
• This work bridges the gap between theoretical analysis and practical application in adaptive optimization.