Norm-Preservation: Why Residual Networks Can Become Extremely Deep?

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Skip connections, notably in Residual Networks (ResNets), enable training of extremely deep neural networks.
• These architectures have demonstrated significant performance improvements in various tasks.

Purpose of the Study:

• To theoretically analyze the role of skip connections in ResNets.
• To investigate the impact of identity skip connections on gradient norm preservation and back-propagation stability.
• To propose methods for enhancing norm-preservation in ResNets.

Main Methods:

• Theoretical analysis of gradient flow in ResNets with identity skip connections.
• Mathematical proofs demonstrating the norm-preserving properties of stacked residual blocks.
• Empirical validation through extensive experiments.
• Development and testing of a novel regularization method for convolution operators.

Main Results:

• Skip connections in residual blocks preserve gradient norms, ensuring stable back-propagation.
• Norm-preservation increases with the depth of stacked residual blocks.
• A new method, Procrustes ResNets, enhances norm-preservation by regularizing singular values.
• Improved learning dynamics and classification performance were observed with Procrustes ResNets.

Conclusions:

• Identity skip connections are crucial for the optimization stability of deep neural networks.
• The proposed Procrustes ResNet modification offers a way to further improve deep network training.
• These findings provide guidance for designing and training even deeper and more effective neural network architectures.