Extended Hamiltonian learning on Riemannian manifolds: theoretical aspects

Area of Science:

• Computational mathematics
• Machine learning theory
• Dynamical systems

Background:

• Learning on Riemannian manifolds is crucial for many scientific applications.
• Existing gradient-based methods have limitations in certain optimization landscapes.
• Second-order dynamics offer potential for improved learning performance.

Purpose of the Study:

• To introduce a general theory of extended Hamiltonian (second-order) learning on Riemannian manifolds.
• To derive coordinate-free dynamical learning equations.
• To compare the theoretical features and convergence of dynamical learning with gradient-based methods.

Main Methods:

• Derivation of dynamical learning equations using the extended-Hamiltonian stationary-action principle.
• Theoretical analysis comparing dynamical learning with gradient-based learning.
• Investigation of learning dynamics on various relevant manifolds.

Main Results:

• A general theory for extended Hamiltonian (second-order) learning on Riemannian manifolds is established.
• Gradient-based learning is shown to be an instance of dynamical learning.
• Classical gradient-based learning with momentum resembles discrete-time dynamical learning.

Conclusions:

• The proposed dynamical learning theory provides a unified framework for understanding and developing advanced learning algorithms.
• The theory offers insights into the convergence properties of different learning approaches.
• The framework is applicable to diverse manifolds, including Stiefel, Grassmann, and SPD matrices.