Improved weight initialization for deep and narrow feedforward neural network.

Area of Science:

• Artificial Intelligence
• Deep Learning
• Neural Networks

Background:

• The Rectified Linear Unit (ReLU) activation function is crucial for modern deep learning models.
• The "dying ReLU" problem, where neurons become inactive, hinders the training of deep neural networks.
• Existing methods struggle with extremely deep and narrow feedforward networks using ReLU.

Purpose of the Study:

• To propose a novel weight initialization method to overcome the "dying ReLU" problem.
• To enhance the training of deep and narrow feedforward neural networks with ReLU activation.
• To improve signal vector propagation within neural networks.

Main Methods:

• Development of a novel weight initialization strategy.
• Analysis of the properties of the proposed initial weight matrix.
• Experimental validation and comparison with existing initialization methods.

Main Results:

• The proposed initialization method effectively addresses the "dying ReLU" issue.
• Demonstrated improved signal vector propagation due to specific matrix properties.
• Experimental results show superior performance compared to current techniques.

Conclusions:

• The novel weight initialization method is effective for training deep neural networks with ReLU.
• This approach offers a promising solution for challenges in deep and narrow network architectures.
• The method facilitates more robust and efficient deep learning model training.