A Robust Initialization of Residual Blocks for Effective ResNet Training Without Batch Normalization

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Batch normalization is a standard component in modern neural networks.
• Practical issues with batch normalization have spurred research into normalization-free architectures.
• Effective training of normalization-free networks remains a challenge.

Purpose of the Study:

• To investigate the role of weight initialization in training ResNet-like normalization-free networks.
• To propose a simple modification to improve initialization for these networks.
• To demonstrate the efficacy of the proposed method on benchmark datasets.

Main Methods:

• A novel, slight modification to the summation operation within ResNet blocks was introduced.
• This modification facilitates correct weight initialization for the entire network.
• The modified architecture was trained and evaluated on CIFAR-10, CIFAR-100, and ImageNet datasets.

Main Results:

• The proposed weight initialization strategy enables successful training of normalization-free ResNet-like networks.
• The modified architecture achieved competitive performance on CIFAR-10, CIFAR-100, and ImageNet.
• No additional regularization or algorithmic changes were required.

Conclusions:

• Weight initialization is a critical factor for normalization-free deep learning models.
• A simple modification to block summation can effectively initialize ResNet-like architectures.
• This approach offers a viable alternative to batch normalization without performance compromise.