Teacher-student complementary sample contrastive distillation.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Knowledge distillation (KD) is crucial for compressing large models into smaller, efficient ones.
• Existing KD methods often overlook teacher supervision effectiveness and student model overconfidence.
• Overconfident predictions by student models can hinder overall performance.

Purpose of the Study:

• To propose a novel knowledge distillation framework, TSCSCD, addressing limitations in current methods.
• To improve the performance of compact student models through enhanced teacher-student interaction.
• To mitigate issues of weak supervision and overconfident predictions in distilled models.

Main Methods:

• Introduced Teacher-Student Complementary Sample Contrastive Distillation (TSCSCD).
• Developed Contrastive Sample Hardness (CSH) to evaluate teacher supervision quality.
• Implemented Supervision Signal Correction (SSC) and Student Self-Learning (SSL) to refine distillation and regularize predictions.

Main Results:

• TSCSCD demonstrated superior performance compared to state-of-the-art knowledge distillation techniques.
• Experiments on four real-world datasets validated the effectiveness of the proposed framework.
• The components CSH, SSC, and SSL collectively contributed to improved model compression and accuracy.

Conclusions:

• TSCSCD offers a significant advancement in knowledge distillation by effectively utilizing teacher knowledge.
• The framework successfully addresses challenges related to supervision quality and student model overconfidence.
• TSCSCD provides a robust approach for developing high-performing compact models.