Mitigating the Vanishing Gradient Problem Using a Pseudo-Normalizing Method

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Activation functions significantly influence neural network performance.
• Derivatives of activation functions can lead to training difficulties like vanishing or exploding gradients.
• Existing methods struggle to balance gradient amplitude for effective deep learning.

Purpose of the Study:

• To introduce pseudo-normalization as a technique to manage gradient amplitudes in deep neural networks.
• To mitigate the vanishing gradient problem while preventing gradient explosion.
• To investigate the impact of this technique on image classification tasks.

Main Methods:

• Proposed pseudo-normalization by dividing gradients by their root mean square.
• Applied gradient amplification every few layers to maintain amplitudes greater than one.
• Utilized interpretability techniques to analyze network predictions.

Main Results:

• Successfully applied pseudo-normalization to deep learning networks with hyperbolic tangent activation for image classification.
• Demonstrated avoidance of vanishing gradient and gradient explosion.
• Observed that networks primarily used image contour information for categorization, unlike typical characteristic-learning networks.

Conclusions:

• Pseudo-normalization is an effective method for improving deep learning training stability.
• The technique offers a novel approach to gradient management in neural networks.
• This method can be a valuable addition to existing deep learning algorithms, particularly for image classification tasks focusing on contours.