Toward Effective Knowledge Distillation: Navigating Beyond Small-Data Pitfall.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Large models trained on extensive datasets show exceptional performance.
• Knowledge Distillation (KD) is used to create compact models for edge devices.
• Existing KD methods may not scale effectively with larger datasets and models.

Purpose of the Study:

• To investigate the effectiveness of current KD methods on large-scale datasets.
• To identify limitations and guide the development of more robust KD techniques.
• To address the "small-data pitfall" in knowledge distillation.

Main Methods:

• Revisiting and evaluating current KD approaches on large-scale datasets.
• Analyzing the knowledge transfer process in KD.
• Proposing and evaluating a new KD method combining vanilla KD with deep supervision.

Main Results:

• Most KD modifications are ineffective on large datasets, exhibiting a "small-data pitfall."
• Incorporating more information into the student model is crucial for effective KD.
• The proposed method combining KD with deep supervision significantly outperforms existing methods.

Conclusions:

• The effectiveness of KD methods is highly dependent on the scale of the dataset.
• Focusing on enhancing information transfer is more critical than altering loss functions for KD.
• The study provides insights for developing consistently effective KD methods for large-scale applications.