SSA-KD: Self-structure-aware knowledge distillation for convolutional neural networks

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Vision

Background:

• Knowledge Distillation (KD) is effective for model compression in Convolutional Neural Networks (CNNs).
• Existing KD methods often use generic student models, causing incompatibility and inefficiency with specific datasets and tasks.
• Adaptive customization of student models remains a challenge in KD.

Purpose of the Study:

• To propose a novel self-structure-aware knowledge distillation (SSA-KD) method.
• To personalize student models by adapting them to specific datasets and tasks.
• To improve the efficiency and reduce the complexity of model compression.

Main Methods:

• Formulating a sub-network from the original teacher model.
• Conducting knowledge distillation between the teacher and personalized student models.
• Employing a structure-aware pruning method for stable student model customization.

Main Results:

• Achieved higher performance and lower complexity compared to previous KD methods.
• Demonstrated effective student model customization with alleviated incompatibility and inefficiency.
• Obtained the highest compression rate on student models with the lowest implementation complexity.

Conclusions:

• The proposed SSA-KD method offers an effective approach for adaptive CNN model compression.
• Personalized student models generated via structure-aware pruning ensure effectiveness and stability.
• This method addresses limitations of universal student models in KD, enhancing practical applications.