Single-Path Bit Sharing for Automatic Loss-Aware Model Compression.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Deep learning models require significant computational resources.
• Network pruning and quantization are key techniques for deep model compression.
• Current methods often perform pruning and quantization separately, leading to suboptimal results.

Purpose of the Study:

• To develop an automatic, loss-aware method for joint network pruning and quantization.
• To address the trade-off challenges between pruning and quantization.
• To reduce the high computational cost associated with searching for optimal compression configurations.

Main Methods:

• Introduced Single-path Bit Sharing (SBS) for unified model pruning and quantization.
• Developed a single-path model encoding all compression configurations.
• Utilized learnable binary gates for joint learning of configurations and model parameters.
• Transformed configuration search into a subset selection problem.

Main Results:

• SBS significantly reduces computation cost while maintaining high performance.
• Achieved a 22.6x Bit-Operation (BOP) reduction on MobileNetV2 with only a 0.1% Top-1 accuracy drop.
• Demonstrated effectiveness on CIFAR-100 and ImageNet datasets.

Conclusions:

• SBS offers an effective and efficient approach to automatic model compression.
• Joint optimization of pruning and quantization is crucial for superior performance.
• The proposed method simplifies configuration search and reduces computational overhead.