Quantization via Distillation and Contrastive Learning.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Deep neural networks (DNNs) require compression for deployment in resource-limited environments.
• Quantization is a key technique for DNN compression, balancing model size and performance.
• Knowledge distillation (KD) can enhance quantization performance by transferring knowledge from high-precision to low-precision networks.

Purpose of the Study:

• To investigate feature-level information loss during knowledge distillation for network quantization.
• To propose a novel quantization method that improves information preservation.
• To enhance the performance of quantized deep neural networks.

Main Methods:

• A novel quantization method combining feature-level distillation and contrastive learning.
• Focus on feature-level network quantization perception to minimize information loss.
• Utilizing the hyperbolic tangent function for smoother gradient estimation during training.

Main Results:

• The proposed method effectively extracts and preserves valuable information during quantization.
• Quantized networks achieve competitive performance compared to their full-precision counterparts.
• Experimental results validate the efficacy of the approach for DNN compression.

Conclusions:

• The novel quantization method enhances the performance of compressed deep neural networks.
• Feature-level distillation and contrastive learning are crucial for preserving information.
• The approach shows significant potential for real-world applications requiring efficient DNNs.