Riemannian Adaptive Optimization Algorithm and its Application to Natural Language Processing.

Area of Science:

• Machine Learning
• Deep Neural Networks
• Optimization Algorithms

Background:

• Deep neural networks (DNNs) require efficient optimization algorithms for parameter tuning.
• Existing Riemannian optimization methods like RAMSGrad have limitations in direct problem-solving and practical implementation.
• Adaptive optimization in Euclidean space (e.g., AMSGrad) is effective but not directly applicable to Riemannian manifolds.

Purpose of the Study:

• To propose a novel Riemannian adaptive optimization algorithm that addresses limitations of existing methods.
• To enhance the direct solvability of Riemannian stochastic optimization problems.
• To enable the use of constant learning rates for practical DNN optimization on Riemannian manifolds.

Main Methods:

• The proposed algorithm extends AMSGrad (Euclidean) and RAMSGrad (Riemannian).
• It directly tackles Riemannian stochastic optimization problems.
• The algorithm is applied to Poincaré embeddings for WordNet noun transitive closure in hyperbolic space.

Main Results:

• The algorithm stably converges to optimal solutions irrespective of initial learning rates.
• It demonstrates faster convergence compared to existing Riemannian optimization algorithms.
• Numerical experiments validate the algorithm's effectiveness on Poincaré embeddings.

Conclusions:

• The developed Riemannian adaptive optimization algorithm offers a practical and efficient solution for DNN parameter optimization.
• It overcomes key challenges faced by previous Riemannian methods, enabling direct problem-solving and constant learning rates.
• The algorithm shows superior performance and stable convergence in complex applications like hyperbolic embeddings.