Self-Distillation for Randomized Neural Networks.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Traditional knowledge distillation (KD) effectively transfers knowledge from large teacher models to smaller student models.
• Standard KD methods are ineffective for randomized neural networks due to their architecture and performance-model size independence.
• Randomized neural networks present unique challenges for performance enhancement via knowledge transfer.

Purpose of the Study:

• To develop an effective knowledge distillation technique for randomized neural networks.
• To improve the performance of randomized neural networks by leveraging their internal predictions.
• To introduce a self-distillation pipeline tailored for the specific characteristics of randomized neural networks.

Main Methods:

• Proposed a self-distillation pipeline where network predictions serve as an additional training target.
• Integrated network predictions with the original target to create a distillation target containing "dark knowledge".
• Extended the method to multi-generation self-distillation and infinite self-distillation (ISD) for randomized neural networks.

Main Results:

• Demonstrated that self-distillation significantly improves the performance of randomized neural networks.
• Showcased the effectiveness of the multi-teacher integration for enhanced knowledge transfer.
• Validated the proposed methods through theoretical analysis and practical experiments on benchmark datasets.

Conclusions:

• Self-distillation is a viable and effective strategy for enhancing randomized neural networks.
• The proposed pipeline offers a novel solution for overcoming KD limitations in these specific network types.
• The theoretical analysis provides a foundation for understanding and further developing self-distillation in randomized neural networks.