Bidirectionally self-normalizing neural networks.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computational Neuroscience

Background:

• Vanishing and exploding gradients are persistent challenges in training deep neural networks.
• Existing practical solutions lack rigorous theoretical underpinnings.
• High-dimensional probability theory offers a potential framework for analysis.

Purpose of the Study:

• To theoretically address the vanishing and exploding gradients problem in neural networks.
• To propose a novel approach for stabilizing signal propagation.
• To validate the proposed methods with empirical evidence.

Main Methods:

• Utilizing high-dimensional probability theory to analyze gradient flow.
• Introducing Gaussian-Poincaré normalized functions as a new class of activation functions.
• Employing orthogonal weight matrices to constrain signal propagation.
• Conducting experiments on synthetic and real-world datasets.

Main Results:

• Demonstrated that sufficient network width, under mild conditions, mitigates vanishing/exploding gradients with high probability.
• Showcased the effectiveness of Gaussian-Poincaré normalization and orthogonal weight matrices in stabilizing forward and backward signal propagation.
• Empirical validation confirmed the theoretical findings on very deep neural networks.

Conclusions:

• The proposed theoretical framework and practical methods effectively solve the vanishing/exploding gradients problem.
• The approach enhances the trainability of very deep neural networks.
• This work provides a rigorous, provable solution to a long-standing challenge in deep learning.