Knowledge Transfer via Decomposing Essential Information in Convolutional Neural Networks.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Knowledge distillation (KD) is a popular technique for compressing large neural networks (CNNs) into smaller ones.
• Existing KD methods often struggle to effectively transfer essential knowledge and are dependent on the spatial dimensions of teacher feature maps.

Purpose of the Study:

• To develop a KD method that transfers knowledge independently of the teacher's feature map spatial shape.
• To improve the performance and efficiency of student neural networks.

Main Methods:

• Feature map decomposition using Singular Value Decomposition (SVD) to extract shape-independent knowledge.
• A multitask learning approach with adaptive constraint adjustment to facilitate effective knowledge transfer.
• Evaluating the method on CIFAR100 and TinyImageNet datasets.

Main Results:

• The proposed KD method achieved a 2.37% performance improvement on CIFAR100.
• The method demonstrated a 2.89% performance improvement on the TinyImageNet dataset.
• Outperformed the current state-of-the-art KD methods on both datasets.

Conclusions:

• The novel KD approach effectively distills essential knowledge irrespective of feature map spatial dimensions.
• The multitask learning strategy enhances the student model's ability to learn from the teacher.
• This method offers a significant advancement in efficient neural network compression and performance improvement.