Improved Residual Network based on norm-preservation for visual recognition.

Area of Science:

• Computer Vision
• Deep Learning
• Neural Network Architectures

Background:

• Residual Networks (ResNets) are powerful convolutional neural network architectures known for enabling deeper and wider networks with performance gains.
• Training very deep networks often faces optimization challenges due to difficulties in information flow and gradient propagation.
• Existing ResNet architectures can be further improved to enhance stability and learning dynamics.

Purpose of the Study:

• To propose architectural refinements for ResNet that improve information flow and gradient preservation.
• To reduce optimization difficulties in training very deep neural networks.
• To enhance accuracy and inference performance through norm-preserving modifications.

Main Methods:

• Introduced architectural refinements to key ResNet components: input stem, downsampling blocks, projection shortcuts, and identity blocks.
• Focused on preserving the norm of the error gradient within residual blocks to ensure stable backpropagation.
• Enforced norm-preservation throughout the network during training to enhance learning dynamics.

Main Results:

• Demonstrated stable backpropagation and reduced optimization difficulties for very deep networks.
• Achieved high accuracy and inference performance across multiple computer vision tasks.
• Verified effectiveness on ImageNet, CIFAR-100, Kinetics-400, MS-COCO, and PASCAL VOC datasets.

Conclusions:

• The proposed architectural refinements enhance the training stability and performance of ResNet models.
• Norm-preservation is crucial for facilitating stable backpropagation and improving generalization in deep networks.
• These modifications offer insights for designing future high-performance neural network architectures.