A Generic Improvement to Deep Residual Networks Based on Gradient Flow.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Residual Networks (ResNets) are foundational in deep learning for image classification.
• Preactivation ResNets often outperform postactivation variants on smaller datasets like CIFAR.
• Performance discrepancies arise on larger benchmarks like ImageNet, necessitating further investigation.

Purpose of the Study:

• To analyze the incongruity in performance between preactivation and postactivation ResNets on CIFAR versus ImageNet.
• To investigate the role of gradient propagation and downsampling in residual network performance.
• To propose modifications to enhance the performance of standard residual architectures on ImageNet.

Main Methods:

• Theoretical analysis of gradient propagation differences between preactivation and postactivation ResNets.
• Empirical evaluation of network performance with modified shortcut connections.
• Comparison of performance improvements across various residual architectures (ResNets, ResNeXts, SE-Nets).

Main Results:

• Postactivation variants facilitate a more diverse gradient composition from deeper layers to earlier layers compared to preactivation variants.
• Downsampling projections in residual architectures were identified as detrimental to performance.
• Replacing downsampling projections with identity-like dense-reshape shortcuts improved ImageNet classification by up to 1.2% without increasing computational cost.

Conclusions:

• The way residual networks handle gradient propagation significantly impacts performance on large-scale datasets.
• Modifying shortcut connections, specifically addressing downsampling, offers a simple yet effective method to boost performance of various residual architectures.
• The findings provide a pathway for optimizing deep convolutional neural networks for enhanced image classification capabilities.