Mirror Descent and Exponentiated Gradient Algorithms Using Trace-Form Entropies

Area of Science:

• Optimization Theory
• Information Geometry
• Machine Learning

Background:

• Mirror Descent (MD) and Generalized Exponentiated Gradient (GEG) are fundamental optimization algorithms.
• Classical methods often struggle with vanishing/exploding gradients and non-Euclidean geometries.
• Generalized entropies offer a flexible framework for defining divergences and metrics.

Purpose of the Study:

• To introduce a unified framework for MD and GEG algorithms based on generalized trace-form entropies.
• To demonstrate improved convergence and robustness properties of these new algorithms.
• To reveal the information-geometric underpinnings connecting these methods to natural gradient descent.

Main Methods:

• Derivation of MD and GEG algorithms from trace-form entropies via deformed logarithms.
• Analysis of convergence behavior and gradient robustness.
• Investigation of connections to Amari's natural gradient and information-geometric structures.
• Application to specific entropy families (Tsallis, Kaniadakis, etc.) to define Riemannian metrics.

Main Results:

• Development of a broad class of MD and GEG algorithms with enhanced convergence and robustness.
• Establishment of a unified geometric foundation for various gradient update rules (additive, multiplicative, natural).
• Demonstration that different entropies induce distinct Riemannian metrics, preserving statistical geometry.
• Tunable parameters allow adaptive geometric selection for improved optimization.

Conclusions:

• The proposed framework unifies first-order optimization methods under generalized Bregman divergences.
• The choice of entropy dictates the underlying Riemannian metric and dual geometric structure.
• These generalized methods offer enhanced adaptability and robustness compared to classical Euclidean optimization.