An effective SteinGLM initialization scheme for training multi-layer feedforward sigmoidal neural networks.

Area of Science:

• Machine Learning
• Artificial Intelligence
• Statistical Modeling

Background:

• Network initialization is crucial for effective neural network training.
• Existing initialization methods can limit model performance and convergence speed.

Purpose of the Study:

• To propose a novel network initialization scheme, SteinGLM, leveraging Stein's identity.
• To enhance the speed and accuracy of neural network training.

Main Methods:

• Utilizing Stein's identity to initialize projection weights in the first hidden layer via eigenvectors.
• Employing forward propagation for subsequent layer initialization.
• Applying generalized linear modeling for output layer weight initialization.

Main Results:

• SteinGLM demonstrates superior performance in numerical experiments.
• The proposed method achieves faster convergence and higher accuracy than popular initialization techniques.

Conclusions:

• SteinGLM offers a significant advancement in neural network initialization.
• This method provides a more efficient and accurate approach to training deep learning models.