Why Is Everyone Training Very Deep Neural Network With Skip Connections?

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Deep neural networks (DNNs) often utilize skip connections to address optimization challenges and enhance generalization.
• Plain networks (PlainNets), lacking skip connections, become untrainable beyond a certain depth, hindering performance.
• The precise mechanisms by which skip connections benefit DNNs remain incompletely understood.

Purpose of the Study:

• To theoretically analyze the role of skip connections in training very deep neural networks.
• To compare the optimization and generalization capabilities of DNNs with and without skip connections.
• To elucidate the mathematical underpinnings of skip connection efficacy using linear algebra and random matrix theory.

Main Methods:

• Theoretical analysis of deep neural networks using concepts from linear algebra and random matrix theory.
• Comparative study of PlainNets against popular skip connection architectures like Residual Networks (ResNets) and ResNeXt.
• Investigation of information flow and representation properties within network layers.

Main Results:

• PlainNets exhibit progressive information loss in hidden representations due to singularity problems as network depth increases, leading to optimization difficulties.
• DNNs with skip connections circumvent singularity issues, preserving information and enabling effective optimization and improved generalization with greater depth.
• Skip connections facilitate the retention of full information, crucial for training deeper and more performant models.

Conclusions:

• Skip connections are critical for overcoming optimization hurdles and enhancing generalization in deep neural networks.
• The theoretical framework confirms that skip connections maintain representational fidelity, enabling the training of significantly deeper models.
• Architectures like ResNets and ResNeXt demonstrate the practical benefits of skip connections in deep learning.