Distillation-Guided Residual Learning for Binary Convolutional Neural Networks.

Area of Science:

• Computer Vision
• Deep Learning
• Artificial Intelligence

Background:

• Binary convolutional neural networks (BCNNs) exhibit a performance gap compared to floating-point CNNs (FCNNs).
• This gap stems from BCNNs' limited modeling capacity and suboptimal training strategies, resulting in significant intermediate feature map residuals.
• Existing BCNNs struggle to match the representational power of FCNNs.

Purpose of the Study:

• To minimize the performance disparity between BCNNs and FCNNs.
• To develop an effective training strategy for BCNNs that enhances their modeling capability.
• To improve the accuracy and efficiency of BCNNs for image classification tasks.

Main Methods:

• Implemented a blockwise distillation loss to enforce similar intermediate feature maps between BCNNs and FCNNs.
• Introduced a lightweight shortcut branch with a squeeze-and-interaction (SI) structure into each binary convolutional block.
• Optimized BCNN architecture to facilitate blockwise distillation, complementing feature map residuals with minimal parameter overhead (<10%).

Main Results:

• The proposed method significantly reduced the performance gap between BCNNs and FCNNs.
• BCNNs trained with the new strategy achieved superior classification accuracy and efficiency on the ImageNet dataset.
• The enhanced BCNN achieved an accuracy of 60.45% on ImageNet, outperforming existing state-of-the-art methods.

Conclusions:

• The combination of blockwise distillation and architectural enhancement effectively bridges the BCNN-FCNN performance gap.
• The lightweight shortcut branch significantly boosts the modeling capability of binary convolutional blocks.
• This approach offers a promising direction for developing highly efficient and accurate BCNNs for real-world applications.