Effective Training of Convolutional Neural Networks With Low-Bitwidth Weights and Activations

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Training deep convolutional neural networks (CNNs) with low-bitwidth weights and activations presents significant challenges.
• The non-differentiability of quantizers in low-precision networks can lead to substantial accuracy degradation.
• Existing methods often struggle to balance network precision and performance effectively.

Purpose of the Study:

• To develop practical and effective approaches for training deep convolutional neural networks with both low-bitwidth weights and activations.
• To mitigate accuracy loss typically associated with network quantization.
• To enhance the efficiency and performance of quantized deep learning models.

Main Methods:

• Progressive quantization: optimizing weights first, then activations, or gradually decreasing bitwidth during training.
• Stochastic precision: a one-stage strategy that randomly quantizes sub-networks while maintaining full precision for others.
• Joint knowledge distillation: training a full-precision model alongside the low-precision model to guide its learning.

Main Results:

• The proposed methods demonstrate effectiveness in improving the training of low-precision networks.
• Experiments on CIFAR-100 and ImageNet datasets validate the efficacy of the developed techniques.
• The joint knowledge distillation approach significantly enhances the performance of the low-precision network.

Conclusions:

• The introduced techniques, including progressive quantization, stochastic precision, and joint knowledge distillation, offer practical solutions for training efficient, low-precision deep convolutional neural networks.
• These methods successfully address the challenges of accuracy loss in quantized networks.
• The findings pave the way for more efficient deployment of deep learning models on resource-constrained devices.