Related Experiment Videos
Accelerating neural network training using weight extrapolations.
1Department of Mechanical, Industrial and Manufacturing Engineering, Northeastern University, 334 Snell Eng. Center, Boston, MA, USA
Summary
A new extrapolation technique accelerates the backpropagation (BP) algorithm for neural networks. This method significantly reduces training time without compromising solution accuracy, offering a practical solution for complex machine learning tasks.
Area of Science:
- Artificial Intelligence
- Machine Learning
- Neural Networks
Background:
- The backpropagation (BP) algorithm is robust for training feedforward neural networks but suffers from long training times.
- Lengthy training processes, involving numerous passes over the complete dataset, can hinder real-world applications of BP.
Purpose of the Study:
- To present a universal acceleration technique for the BP algorithm.
- To introduce a method that significantly reduces computation time while maintaining solution quality.
Main Methods:
- A novel extrapolation technique is applied to individual interconnection weights during BP training.
- This procedure is implemented sparingly, activated only a few times between conventional BP iterations.
- The method minimally alters the computational structure of the standard BP algorithm.
Main Results:
- Demonstrated significant savings in computation time across three example applications.
- The computed solutions remained in close proximity to those obtained by the conventional BP procedure.
- The proposed method accelerates BP without degrading the usefulness of the solutions.
Conclusions:
- The presented extrapolation technique offers a viable and effective acceleration for the BP algorithm.
- This approach provides substantial computational savings, making BP more practical for demanding applications.
- The method's performance is comparable to or better than other advanced techniques like the conjugate gradient algorithm.