Self-Distillation: Towards Efficient and Compact Neural Networks.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Vision

Background:

• Deep neural networks (DNNs) achieve high accuracy but suffer from large computation and parameter requirements, hindering deployment.
• Existing model compression techniques face limitations in balancing accuracy and efficiency.

Purpose of the Study:

• To introduce a novel knowledge distillation technique, self-distillation, to improve DNN accuracy and reduce computational costs.
• To enable efficient deployment of deep learning models without compromising performance.

Main Methods:

• Self-distillation integrates attention modules and shallow classifiers into DNNs.
• Knowledge is distilled from deeper layers to shallower layers within the same network.
• Dynamic network operation is enabled through additional classifiers for acceleration.

Main Results:

• Self-distillation consistently improves accuracy across various DNNs and datasets.
• Average accuracy gains of 3.49% on CIFAR100 and 2.32% on ImageNet were observed.
• The technique is compatible with other model compression methods like pruning and lightweight design.

Conclusions:

• Self-distillation offers an effective solution for deploying accurate and efficient deep neural networks.
• The method presents a promising approach for advancing model compression and deployment in AI.
• Further research can explore combinations with other optimization techniques for enhanced results.