A Comprehensive and Modularized Statistical Framework for Gradient Norm Equality in Deep Neural Networks.

Area of Science:

• Deep Learning
• Neural Network Optimization
• Machine Learning Metrics

Background:

• Deep neural networks (DNNs) development is hindered by gradient explosion and vanishing issues.
• Existing metrics for these issues have limitations due to network complexity and require strong assumptions.
• There is a need for more robust and broadly applicable metrics for DNN training stability.

Purpose of the Study:

• To propose a novel metric, Block Dynamical Isometry, for assessing gradient norm changes in individual network blocks.
• To develop a modular statistical framework using free probability to handle complex DNN architectures.
• To analyze and improve existing deep learning optimization techniques and introduce new ones.

Main Methods:

• Introduced Block Dynamical Isometry (BDI) metric inspired by Gradient Norm Equality and dynamical isometry.
• Developed a free probability-based statistical framework for analyzing serial-parallel hybrid connections and diverse network components.
• Analyzed initialization, normalization, and network structures using the proposed metric and framework.

Main Results:

• Block Dynamical Isometry was found to be a universal principle underlying various optimization techniques.
• Improvements were made to activation function selection, weight normalization, SeLU coefficient derivation, and DenseNet optimization.
• A novel Second Moment Normalization technique was introduced, offering reduced computation and better performance, especially with small batch sizes.

Conclusions:

• Block Dynamical Isometry provides a unified perspective on gradient stability in DNNs.
• The developed framework effectively analyzes complex network architectures.
• Novel methods and improvements enhance DNN training efficiency and performance, validated through extensive experiments.