Multi-way backpropagation for training compact deep neural networks.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Deep convolutional neural networks (CNNs) benefit from increased depth.
• Skip connections in architectures like ResNet mitigate vanishing gradients.
• Very deep CNNs can experience supervision vanishing in intermediate layers due to long backpropagation paths, weakening their representation power.

Purpose of the Study:

• Investigate the supervision vanishing issue in existing backpropagation (BP) methods.
• Propose an effective method, Multi-way BP (MW-BP), to address supervision vanishing.
• Enhance the performance and compactness of deep CNNs.

Main Methods:

• Introduce multiple auxiliary losses to intermediate layers of deep neural networks.
• Apply the Multi-way BP (MW-BP) method to various deep architectures (e.g., ResNet, MobileNet).
• Evaluate model performance and compactness against existing methods and model compression techniques.

Main Results:

• MW-BP effectively addresses the supervision vanishing issue in deep CNNs.
• Resultant models (e.g., MwResNet-44) are more compact and perform better than deeper conventional models (e.g., ResNet-110).
• Achieved superior performance compared to state-of-the-art model compression methods.
• Demonstrated effectiveness in image classification and face recognition tasks.

Conclusions:

• MW-BP offers a viable solution to supervision vanishing in deep CNNs.
• The method yields significantly more compact and performant models.
• MW-BP facilitates inherent model selection by producing multiple compact models of varying depths.