Densely Distilled Flow-Based Knowledge Transfer in Teacher-Student Framework for Image Classification.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Deep neural networks (DNNs) require extensive training data and computational resources.
• Knowledge transfer methods aim to leverage pre-trained models to improve training efficiency and performance of new models.
• Existing teacher-student frameworks (TSFs) have limitations in effectively distilling and transferring complex knowledge across layers.

Purpose of the Study:

• To propose a new teacher-student framework (TSF)-based knowledge transfer method.
• To enhance the efficiency and accuracy of knowledge transfer between deep neural networks.
• To investigate a dense flow-based sequential knowledge transfer scheme.

Main Methods:

• A novel TSF employing dense flow-based knowledge distillation from a pre-trained teacher DNN to a student DNN.
• Extraction of multiple overlapped, flow-based information items across layers from the teacher DNN.
• Repetitive, bottom-up sequential training between teacher and student DNNs for knowledge transference.
• Evaluation on benchmark image datasets: CIFAR-10, CIFAR-100, MNIST, and SVHN.

Main Results:

• The proposed dense flow-based sequential knowledge transfer scheme significantly improves student DNN accuracy.
• The trained student ResNet model more accurately reflects the rich information of the pre-trained teacher ResNet.
• Superior image classification accuracy compared to existing TSF-based knowledge transfer methods across all tested datasets.
• Demonstrated fast optimization capabilities.

Conclusions:

• The proposed TSF with dense flow-based sequential knowledge transfer is effective for improving DNN performance.
• This method offers a more efficient and accurate way to transfer knowledge compared to existing approaches.
• The framework shows promise for applications requiring reliable image classification and optimized training.