Improving the convergence of the backpropagation algorithm using learning rate adaptation methods.
G D Magoulas1, M N Vrahatis, G S Androulakis
1Department of Informatics, University of Athens, GR-157.71, Athens, Greece.
Neural Computation
|September 22, 1999
Summary
This study introduces new gradient-based backpropagation algorithms with adaptive learning rates and line search, improving training speed and stability for neural networks.
Area of Science:
- Artificial Intelligence
- Machine Learning
- Neural Networks
Background:
- Backpropagation algorithms are fundamental for training neural networks.
- Existing methods face challenges in convergence rate and stability.
- Adaptive learning rates and line search techniques can potentially enhance training efficiency.
Purpose of the Study:
- To develop and evaluate novel gradient-based backpropagation algorithms.
- To improve convergence rate and stability in neural network training.
- To compare proposed methods against popular training techniques.
Main Methods:
- Utilized gradient-based backpropagation with adaptive learning rates (common or individual).
- Incorporated Goldstein/Armijo line search for efficient step size determination.
- Estimated local Lipschitz constants without extra function or gradient evaluations.
Main Results:
- Proposed algorithms demonstrated improved convergence rates.
- Enhanced learning stability and robustness against oscillations were observed.
- Simulations confirmed superior performance compared to several popular training methods.
Conclusions:
- The novel adaptive learning rate algorithms offer significant improvements for backpropagation.
- These methods provide a more stable and efficient approach to training neural networks.
- The findings suggest a promising direction for optimizing deep learning model training.
