Distilling structural knowledge from CNNs to vision transformers for data-efficient visual recognition

Knowledge distillation (KD) is an effective strategy to transfer learned representations from a pre-trained teacher model to a smaller student model. Current methods for knowledge transfer from convolutional neural networks (CNNs) to vision transformers (ViTs) mainly align output logits. However, such approaches often overlook the rich semantic structures encoded in CNN features, thereby restricting ViTs from effectively inheriting the inductive biases inherent in convolutional architectures. To this end, this paper proposes a Feature-based CNN-to-ViT Structural Knowledge Distillation framework, dubbed FSKD, which combines the semantic structural knowledge embedded in CNN (teacher) features with the strength of ViT (student) in capturing long-range dependencies. Specifically, this framework includes a feature alignment module to bridge the representational gap between CNN and ViT features, and it incorporates a global feature alignment loss. Additionally, we develop patch-wise and attention-wise distillation losses to transfer inter-patch similarity and attention distribution, facilitating semantic structural knowledge transfer from CNNs to ViTs. Experimental results demonstrate that the proposed method considerably enhances ViT performance in visual recognition tasks, particularly under scenarios with limited data. Code is available at Github.