Globally convergent algorithms with local learning rates.

Area of Science:

• Machine Learning
• Optimization Theory

Background:

• First-order batch training algorithms with local learning rates often lack guaranteed convergence.
• Ensuring convergence from remote initial weights is a significant challenge in machine learning optimization.

Purpose of the Study:

• To present a novel generalized theoretical result for developing globally convergent first-order batch training algorithms.
• To equip existing algorithms with a strategy for adapting search direction to ensure descent.
• To demonstrate guaranteed convergence to a local minimizer of the batch error function.

Main Methods:

• Development of a generalized theoretical framework for first-order batch training.
• Incorporation of a strategy to adapt the search direction to a descent direction.
• Empirical validation through application examples comparing standard and modified algorithms.

Main Results:

• A novel theoretical result is established that underpins globally convergent algorithms.
• Algorithms are equipped with adaptive search direction strategies, ensuring error decrease per iteration.
• Convergence to a local minimizer is achieved even from remote initial weights.

Conclusions:

• The presented theoretical result effectively enhances the convergence properties of first-order batch training algorithms.
• The modified algorithms demonstrate robust performance, achieving guaranteed convergence in application examples.
• This work provides a foundation for developing more reliable and efficient machine learning training methods.