A study on the optimal double parameters for steepest descent with momentum

Area of Science:

• Optimization Algorithms
• Numerical Analysis
• Machine Learning Theory

Background:

• Steepest descent algorithms are fundamental in optimization.
• Momentum is often used to accelerate convergence.
• Previous work identified local optimal parameters for these algorithms.

Purpose of the Study:

• To extend existing local optimal parameters to global optimal parameters.
• To analyze the stability of steepest descent algorithms with momentum.
• To achieve the fastest possible convergence for quadratic functions.

Main Methods:

• Stability analysis of two steepest descent algorithms with momentum.
• Extension of local optimal parameters (Torii & Hagan, 2002; Zhang, 2013) to global optimal parameters.
• Simultaneous determination of optimal learning rates and momentum factors.

Main Results:

• Global optimal parameters were successfully derived.
• The derived parameters ensure the fastest convergence.
• Stability analysis confirmed the effectiveness of the extended parameters.

Conclusions:

• Simultaneously optimizing learning rates and momentum factors leads to global optima.
• This approach guarantees the fastest convergence for steepest descent algorithms on quadratic functions.
• The findings provide a theoretical foundation for accelerated optimization.