Distilling structural knowledge from CNNs to vision transformers for data-efficient visual recognition

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Knowledge distillation (KD) transfers model representations, typically aligning output logits.
• Existing methods for CNN-to-ViT transfer overlook rich semantic structures in CNN features.
• This limits Vision Transformers (ViTs) in inheriting convolutional neural network (CNN) inductive biases.

Purpose of the Study:

• Propose a Feature-based CNN-to-ViT Structural Knowledge Distillation (FSKD) framework.
• Integrate semantic structural knowledge from CNN features with ViT's long-range dependency capabilities.
• Enhance ViT performance in visual recognition, particularly in low-data regimes.

Main Methods:

• Develop a feature alignment module to bridge CNN and ViT representational gaps.
• Incorporate a global feature alignment loss.
• Introduce patch-wise and attention-wise distillation losses for inter-patch similarity and attention distribution transfer.

Main Results:

• FSKD effectively transfers semantic structural knowledge from CNNs to ViTs.
• The framework significantly improves ViT performance in visual recognition tasks.
• Performance gains are particularly notable in scenarios with limited training data.

Conclusions:

• FSKD offers a novel approach to knowledge distillation from CNNs to ViTs.
• The method successfully transfers rich structural information beyond simple logit alignment.
• FSKD demonstrates the potential for improved ViT generalization and efficiency, especially in data-scarce environments.