Learning Student Networks via Feature Embedding.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Deep convolutional neural networks (CNNs) offer powerful capabilities but require significant computational resources, limiting their deployment on mobile devices.
• Knowledge distillation is a technique to transfer knowledge from a large teacher network to a smaller, more portable student network.

Purpose of the Study:

• To develop a knowledge distillation method that avoids auxiliary parameters, reducing storage and computational complexity.
• To enable the deployment of high-performance CNNs on resource-constrained mobile devices.

Main Methods:

• A novel teacher-student learning paradigm focusing on feature embedding.
• Introduction of a locality preserving loss to guide the student network in learning low-dimensional features that retain intrinsic properties.
• Theoretical analysis to demonstrate reduced computational complexity.

Main Results:

• The proposed method successfully propagates knowledge from teacher to student networks without introducing additional optimizable parameters.
• The student network generates low-dimensional features that inherit properties from the teacher's high-dimensional features.
• Experimental results show superior performance compared to existing teacher-student methods in terms of computational and storage efficiency.

Conclusions:

• The feature embedding approach with locality preserving loss offers an efficient knowledge distillation solution.
• This method effectively creates portable networks that maintain high performance while significantly reducing resource requirements.
• The technique is validated through experiments on benchmark datasets and well-trained networks.