Multi-Objective Convex Quantization for Efficient Model Compression.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Vision

Background:

• Model compression is crucial for efficient deep learning deployment.
• Existing quantization methods face challenges due to the non-differentiability of quantization operations.
• Single-objective optimization struggles to balance accuracy and quantization constraints.

Purpose of the Study:

• To propose a novel multi-objective convex quantization method for efficient model compression.
• To address the non-differentiability issue in network quantization during training.
• To achieve a balance between high network precision and low quantization error.

Main Methods:

• Modeled network training as a multi-objective optimization problem.
• Designed a differentiable quantization error function ensuring computational convexity.
• Implemented a time-series self-distillation training scheme.
• Introduced dynamic Lagrangian coefficient adaptation to balance losses.

Main Results:

• Successfully integrated quantization into network training by avoiding non-differentiable back-propagation.
• Achieved controllable and stable performance convergence through self-distillation.
• Demonstrated outstanding performance on benchmarks like MNIST, CIFAR-10/100, ImageNet, Penn Treebank, and Microsoft COCO.
• Outperformed existing model compression methods.

Conclusions:

• The proposed multi-objective convex quantization effectively compresses models while maintaining high performance.
• The novel training scheme and coefficient adaptation enable stable and efficient optimization.
• This method offers a significant advancement in deep learning model compression.