Theoretical Convergence Analysis and Initialization Comparisons of Deep Soft-Thresholding Networks.

Area of Science:

• Deep Learning
• Neural Network Architectures
• Computational Neuroscience

Background:

• Soft-thresholding (ST) is integral to deep neural networks, particularly deep soft-thresholding fully connected networks (ST-FCNs).
• Training deep ST-FCNs often faces convergence issues, including time-consuming training and gradient explosion, due to an incomplete understanding of their convergence behavior.

Purpose of the Study:

• To establish a theoretical link between the convergence of deep ST-FCNs and their weight and bias parameters.
• To propose an effective initialization strategy for enhancing ST-FCN convergence.

Main Methods:

• Theoretical analysis to determine the convergence conditions for deep ST-FCNs as the number of layers approaches infinity.
• Empirical evaluation comparing identity matrix initialization against standard methods (Gaussian, He, LeCun, Xavier, Uniform) on synthetic and real-world datasets.
• Validation across various network depths, architectures (DenseNet, ResNet, VGG), and challenging benchmarks.

Main Results:

• Theoretical analysis indicates deep ST-FCNs converge when weights approach an identity matrix and biases approach zero.
• Identity matrix initialization for weights and zero for biases demonstrated significantly faster and more stable convergence compared to other methods.
• Findings were robust across diverse datasets (MNIST, CIFAR-10/100, STL-10, Tiny ImageNet) and network complexities.

Conclusions:

• Initializing deep ST-FCNs with identity matrix weights and zero biases provides a theoretically grounded and empirically validated method for rapid and stable convergence.
• This research offers a foundational understanding of ST neural network convergence and extends to recurrent neural networks (RNNs).