An Innovative Multisource Teacher Collaborative Framework for Self-Knowledge Distillation.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Traditional knowledge distillation (KD) relies on pretrained teachers, which is computationally intensive.
• Self-knowledge distillation (SKD) learns from its own predictions for efficiency.
• Existing SKD methods often overlook the benefits of integrating diverse knowledge sources.

Purpose of the Study:

• To propose a novel multisource teacher collaboration for self-knowledge distillation (MSTCS-KD) framework.
• To enhance model learning capability by integrating complementary knowledge from diverse teacher types.
• To improve feature learning, generalization, and trainability in deep learning models.

Main Methods:

• Implemented lightweight auxiliary branches with varied structures in the student network.
• Incorporated a teacher-guided attention mechanism for adaptive learning.
• Performed collaborative distillation by combining heterogeneous and homogeneous knowledge from different network layers and augmented samples.

Main Results:

• MSTCS-KD demonstrated superior performance compared to state-of-the-art SKD methods.
• The proposed method showed significant improvements across various network architectures and datasets.
• The complementary distillation approach effectively enhanced feature learning and generalization.

Conclusions:

• Multisource teacher collaboration offers a promising direction for advancing self-knowledge distillation.
• MSTCS-KD effectively leverages diverse knowledge sources to boost model performance.
• The method provides a more robust and efficient approach to model training.