Random Orthogonal Additive Filters: A Solution to the Vanishing/Exploding Gradient of Deep Neural Networks

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• The vanishing/exploding (V/E) gradient problem has hindered neural network (NN) training since the early 1990s.
• Existing solutions have not fully resolved this fundamental obstacle in deep learning.

Purpose of the Study:

• To develop a novel NN architecture that overcomes the V/E gradient issue.
• To achieve stable training for extremely deep neural networks and improve performance on tasks with long-term dependencies.

Main Methods:

• The study proposes an architecture based on approximate dynamical isometry, where singular values of the input-output Jacobian (IOJ) are centered around 1.
• This involves filtering previous activations orthogonally and combining them with nonlinear activations of the next layer, creating a convex combination.
• Analytical bounds demonstrate the impossibility of gradient vanishing or exploding, even for infinite-depth networks.

Main Results:

• Training of a 50,000-layer multilayer perceptron (MLP) and an Elman NN for 10,000 time steps was successfully demonstrated.
• The proposed model shows superior performance and simplicity compared to architectures like LSTMs.
• A single-layer recurrent neural network (RNN) enhanced with this method achieved state-of-the-art results, reaching over 98% accuracy on the psMNIST task within ten epochs.

Conclusions:

• The novel architecture effectively solves the V/E gradient problem, enabling unprecedented network depths.
• This approach offers a simpler and more effective alternative to existing methods for handling long-term dependencies.
• The findings pave the way for more efficient and powerful deep learning models.