Related Experiment Videos
Deterministic convergence of an online gradient method for BP neural networks
Wei Wu1, Guorui Feng, Zhengxue Li
1Applied Mathematics Department, Dalian University of Technology, Dalian 116023, China. wuweiw@dlut.edu.cn
IEEE Transactions on Neural Networks
|June 9, 2005
Summary
This study introduces a deterministic and monotone convergence theorem for online gradient methods in backward propagation (BP) neural networks with a hidden layer, advancing training stability.
Area of Science:
- Machine Learning
- Artificial Intelligence
- Neural Networks
Background:
- Online gradient methods are prevalent for training feedforward neural networks.
- Existing convergence results for these methods are often probabilistic and nonmonotone.
Purpose of the Study:
- To establish a novel convergence theorem for online gradient methods.
- To specifically address backward propagation (BP) neural networks with a hidden layer.
- To demonstrate a deterministic and monotone convergence property.
Main Methods:
- Development of a theoretical framework for analyzing online gradient descent.
- Application of the framework to backward propagation (BP) neural networks.
- Mathematical proof of a convergence theorem for variable step-size algorithms.
Main Results:
- A convergence theorem for online gradient methods in BP neural networks with a hidden layer is proven.
- The established convergence is shown to be deterministic.
- The convergence is also demonstrated to be monotone, unlike many existing results.
Conclusions:
- The findings offer a more predictable and stable convergence guarantee for training specific neural network architectures.
- This deterministic and monotone convergence is a significant theoretical advancement.
- The results have implications for the reliable training of deep learning models.