Related Experiment Videos
Improved backpropagation learning in neural networks with windowed momentum
1Brigham Young University, Computer Science Department, 3361 TMCB, P.O. Box 26576, Provo Utah 84602-6576, USA. butch@axon.cs.byu.edu
International Journal of Neural Systems
|October 9, 2002
Summary
Windowed Momentum, a new neural network training algorithm, significantly speeds up convergence time by using recent weight updates. This method achieves an average 32% speedup while maintaining or improving accuracy, even with varied data presentation orders.
Area of Science:
- Artificial Intelligence
- Machine Learning
- Neural Networks
Background:
- Backpropagation training in neural networks can be time-consuming.
- Momentum is a standard technique to accelerate convergence and maintain generalization.
- Optimizing neural network training speed is crucial for practical applications.
Purpose of the Study:
- To introduce and evaluate the Windowed Momentum algorithm.
- To demonstrate its speedup capabilities compared to Standard Momentum.
- To assess its performance across various datasets and its robustness to sample presentation order.
Main Methods:
- Developed the Windowed Momentum algorithm, utilizing a fixed-width history of recent weight updates.
- Tested the algorithm on 15 diverse datasets, including a large Optical Character Recognition (OCR) dataset.
- Compared convergence times and accuracy against Standard Momentum.
Main Results:
- Windowed Momentum achieved an average convergence speedup of 32% across 15 datasets.
- The algorithm maintained or improved accuracy compared to Standard Momentum.
- Windowed Momentum demonstrated resilience to negative effects of poor sample presentation order.
Conclusions:
- Windowed Momentum offers significant speedup in neural network training convergence.
- The algorithm provides a robust alternative to Standard Momentum, handling data order variations effectively.
- This technique enhances the efficiency of neural network training without compromising accuracy.