Diverse Sample Generation: Pushing the Limit of Generative Data-Free Quantization.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Generative data-free quantization compresses deep neural networks (DNNs) to low bit-widths without real data access.
• This method uses batch normalization (BN) statistics from full-precision networks to generate synthetic data for quantization.
• A key challenge is accuracy degradation due to the homogenization of generated synthetic data.

Purpose of the Study:

• To address the accuracy degradation in generative data-free quantization.
• To introduce a novel scheme for diversifying synthetic data generation.
• To improve the effectiveness of data-free quantization techniques.

Main Methods:

• Theoretical analysis revealed that synthetic sample diversity is crucial for data-free quantization.
• Proposed a generic Diverse Sample Generation (DSG) scheme to mitigate synthetic data homogenization.
• DSG relaxes BN statistics alignment, strengthens BN layer loss impact per sample, and inhibits inter-sample correlation.

Main Results:

• The DSG scheme effectively mitigates detrimental homogenization in synthetic data generation.
• Comprehensive experiments on large-scale image classification demonstrate consistent performance gains across various neural architectures, especially at ultra-low bit-widths.
• Data diversification via DSG provides general improvements to quantization-aware training and post-training quantization methods.

Conclusions:

• The proposed Diverse Sample Generation (DSG) scheme is a general and effective method for improving generative data-free quantization.
• DSG enhances quantization performance by diversifying synthetic data, overcoming the limitations of existing BN statistics-constrained approaches.
• This work highlights the importance of synthetic data diversity for successful ultra-low bit-width neural network compression.