Dual-Level Knowledge Distillation via Knowledge Alignment and Correlation.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Vision

Background:

• Knowledge distillation (KD) is crucial for model compression and knowledge transfer.
• Standard KD methods indirectly align knowledge via class prototypes, ignoring inter-sample structural knowledge (knowledge correlation).
• Existing contrastive learning-based distillation methods struggle with correlation objectives, negatively impacting performance.

Purpose of the Study:

• To introduce a novel knowledge correlation objective to improve distillation performance.
• To propose Dual-Level Knowledge Distillation (DLKD), integrating knowledge alignment and correlation explicitly.
• To demonstrate the necessity of both alignment and correlation for effective distillation.

Main Methods:

• Developed a novel knowledge correlation objective.
• Introduced Dual-Level Knowledge Distillation (DLKD) combining explicit knowledge alignment and correlation.
• Evaluated DLKD's task-agnostic and model-agnostic properties for knowledge transfer from pretrained teachers to students.

Main Results:

• DLKD significantly outperforms state-of-the-art methods across diverse experimental settings.
• Knowledge correlation acts as an effective regularization for learning generalized representations.
• The proposed method enables effective knowledge transfer from both supervised and self-supervised pretrained teachers.

Conclusions:

• DLKD offers a superior approach to knowledge distillation by explicitly addressing both knowledge alignment and correlation.
• The integration of knowledge correlation enhances representation generalization and overall distillation effectiveness.
• DLKD provides a versatile and powerful framework for model compression and knowledge transfer.