Two-phase collaborative model compression training for joint pruning and quantization.

Area of Science:

• Artificial Intelligence
• Computer Science
• Machine Learning

Background:

• Neural network model compression is crucial for reducing storage and computational demands.
• Existing compression methods like pruning and quantization often lack effective integration, limiting performance.
• A unified approach is needed to leverage the benefits of diverse compression techniques simultaneously.

Purpose of the Study:

• To propose a novel two-phase collaborative training framework for joint pruning and quantization.
• To achieve synergistic optimization of multiple neural network compression techniques.
• To enhance model compression efficiency and accuracy by integrating pruning and quantization.

Main Methods:

• A two-phase framework: collaborative constraint pre-compression and post-training compression refinement.
• Unified constraint loss function for weight proximity to quantization values.
• Sparse regularization for automated network structure learning in pruning.
• Iterative optimization to minimize quantization errors and achieve 2^n quantization.

Main Results:

• Demonstrated significant reduction in network parameters with considerable accuracy preservation on MNIST, CIFAR-10, and CIFAR-100 datasets.
• Achieved excellent effectiveness in both compression ratio and accuracy.
• The framework successfully integrated pruning and quantization, minimizing adverse interactions.

Conclusions:

• The proposed framework effectively integrates pruning and quantization for synergistic model compression.
• It offers a viable solution for developing more efficient neural network models suitable for hardware implementation.
• The approach minimizes negative impacts between compression techniques, enhancing overall performance.