Long-range zero-shot generative deep network quantization.

Area of Science:

• Deep Learning
• Model Quantization
• Generative Models

Background:

• Quantization reduces deep network computational cost by using low bit-width numbers.
• Zero-shot quantization synthesizes data to approximate real data distributions without original data access.
• Existing zero-shot methods suffer from limited data diversity and feature richness.

Purpose of the Study:

• To propose a novel deep network quantizer, long-range zero-shot generative deep network quantization (LRQ).
• To address the limitations of current zero-shot quantization techniques.
• To improve the performance of quantized deep networks without access to real data.

Main Methods:

• Introduced a long-range generator (LRG) incorporating long-range attention and large-kernel convolution for enhanced global feature learning.
• Developed an adversarial margin add (AMA) module to enlarge intra-class angular separation.
• Utilized decoupled knowledge distillation to transfer knowledge from full-precision networks.

Main Results:

• The proposed LRQ method demonstrated superior performance compared to existing quantization techniques.
• LRG effectively captures long-range information, leading to higher synthetic data diversity.
• AMA module successfully improved feature distinctiveness and intra-class heterogeneity.

Conclusions:

• LRQ offers a significant advancement in zero-shot quantization by generating high-quality synthetic data.
• The combination of LRG and AMA modules effectively overcomes the performance gap in zero-shot quantization.
• LRQ provides a viable solution for efficient deep network deployment in resource-constrained environments.