Successfully and efficiently training deep multi-layer perceptrons with logistic activation function simply requires...

Area of Science:

• Deep Learning
• Neural Network Architectures
• Computational Neuroscience

Background:

• The vanishing gradient problem hinders deep neural network training with sigmoid activation functions.
• Standard back-propagation algorithms struggle with deep networks due to diminishing gradients.

Purpose of the Study:

• Investigate the role of weight initialization in the vanishing gradient problem.
• Develop theoretical models to understand gradient behavior during back-propagation.
• Identify conditions to mitigate vanishing gradients in deep neural networks.

Main Methods:

• Developed a theoretical model analyzing the effect of initial weight means on gradient values.
• Formulated a second theoretical model to establish sufficient conditions for vanishing gradients.
• Conducted experiments on 10- and 15-hidden-layer networks to validate theoretical predictions.

Main Results:

• Initial weight mean negatively impacts gradient values.
• A sufficient condition for avoiding vanishing gradients was identified.
• Negative initial weights, inversely proportional to layer neuron count, prevent gradient vanishing.
• Theoretical predictions were experimentally confirmed.

Conclusions:

• Weight initialization is a critical factor in overcoming the vanishing gradient problem.
• Specific initialization strategies enable effective training of deep neural networks with sigmoid activations.
• The findings facilitate more efficient and successful training of deep learning models.