DCCD: Reducing Neural Network Redundancy via Distillation.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Deep neural networks offer high performance but are challenging to deploy on resource-limited devices.
• Knowledge Distillation (KD) compresses large models by transferring knowledge from teacher to student networks.
• Existing KD methods often overlook information redundancy within student networks.

Purpose of the Study:

• To introduce a novel distillation framework, Difference-based Channel Contrastive Distillation (DCCD), for redundancy reduction in student networks.
• To enhance student networks' feature expression and sensitivity to dynamic changes.
• To improve model compression and acceleration for efficient deployment.

Main Methods:

• DCCD incorporates channel contrastive knowledge and dynamic difference knowledge for redundancy reduction.
• A contrastive objective at the feature level broadens the student network's feature expression space.
• Multiview augmented responses are used to extract detailed knowledge and enhance sensitivity to minor changes.

Main Results:

• The student network gains contrastive and difference knowledge, reducing overfitting and redundancy.
• The proposed method achieves surprising results, with student models approaching or exceeding teacher accuracy on CIFAR-100.
• Significant reductions in top-1 error achieved on ImageNet classification (28.16%) and cross-model transfer (24.15%).

Conclusions:

• DCCD effectively reduces redundancy and overfitting in student networks through contrastive and difference knowledge.
• The framework demonstrates state-of-the-art accuracy compared to other distillation methods.
• DCCD offers a promising approach for deploying high-performance deep neural models on resource-constrained devices.