Toward Enhancing RMSProp With Forward-Looking Gradient Updates for Complex Loss Landscapes.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Training deep neural networks with many nonlinear layers presents challenges.
• Traditional gradient methods struggle with models lacking skip connections, which focus on high-order feature extraction.

Purpose of the Study:

• Introduce a novel algorithm for efficient deep neural network training.
• Improve learning efficiency in deep models with numerous nonlinear layers and no residual structure.

Main Methods:

• Developed an approximated integrated gradient averaged over the weight update range.
• Integrated the approximated average gradient into the RMSProp optimizer.
• Evaluated the novel algorithm against conventional RMSProp and Adam on benchmark datasets.

Main Results:

• The novel algorithm significantly reduced iterations needed to reach target training loss on MNIST, Fashion MNIST, and IMDb benchmarks.
• Demonstrated effectiveness across convolutional and fully connected architectures with varying initializations.
• Performance on shallow models remained comparable to standard RMSProp, with moderate increases in computational and memory costs.

Conclusions:

• The average gradient concept offers an efficient alternative to high-order derivative computation for deep learning.
• The proposed algorithm enhances RMSProp efficiency for specific deep model architectures.
• This approach shows promise for training complex neural networks, particularly those mimicking biological brain structures.