Mixed precision quantization based on information entropy.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Mixed precision quantization reduces computational and memory demands by decreasing model bit width.
• Improper bit-width allocation in quantization can lead to wasted resources and reduced model performance.

Purpose of the Study:

• To propose an adaptive bit-width allocation method for mixed precision quantization using information entropy.
• To mitigate precision loss and optimize resource utilization during model compression.
• To automate bit-width allocation while maintaining high model accuracy.

Main Methods:

• Calculating layer output entropy during the forward pass and smoothing values with a sliding window.
• Dynamically determining a bit-width threshold based on smoothed average entropy for adaptive layer-wise allocation.
• Utilizing Optuna for hyperparameter optimization (threshold and window size) with model accuracy as a constraint.
• Integrating knowledge distillation with a larger teacher model to guide the training of the quantized model.

Main Results:

• Successfully reduced model bit width for weights and activations to 3.6M/3.6MP on ResNet architectures.
• Achieved minimal accuracy loss (max 0.6%) on CIFAR-100 and comparable accuracy to full-precision models on CIFAR-10.
• Demonstrated effective balancing of model compression and performance.

Conclusions:

• The proposed information entropy-based method effectively optimizes mixed precision quantization.
• Adaptive bit-width allocation significantly reduces model size and computational requirements without compromising accuracy.
• Knowledge distillation further enhances the performance of the compressed models.