SSA-KD: Self-structure-aware knowledge distillation for convolutional neural networks

Knowledge Distillation has achieved great success in model compression for convolutional neural networks. However, the selection of the student model usually relies on universal small structures, which leads to plentiful incompatibility and inefficiency, i.e., the student model cannot be customized adaptively in terms of specified datasets and tasks. In this paper, we propose a self-structure-aware knowledge distillation to obtain the student model by personalizing the teacher model, namely, we first formulate a sub-network from the original teacher model, and then conduct the knowledge distillation across the teacher and student models. The customization of the student model is finalized via a structure-aware pruning method, which can yield a stable structure to ensure the effectiveness of the student model. Compared with previous knowledge distillation methods, our method can be executed with lower complexity while with the higher performance because the structure-aware pruning method can generate a layer-aligned sub-structure from the teacher model. It means the incompatibility and inefficiency concerns can be alleviated under the appropriate customization for the student model. We test the method on VGG-16, ResNet-32, and ResNet-50 with CIFAR-10 and CIFAR-100. Not only can we acquire the highest compression rate on the student model, but also require the lowest complexity to implement the method due to the same depth dimension between the teacher and the student model. Code is available at: https://github.com/motinwing/AFIE-distillation-SSA-KD.