Neighborhood relation-based knowledge distillation for image classification.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Knowledge distillation is a key model compression technique, transferring knowledge from large teacher models to smaller student models.
• Existing relational distillation methods often overlook the nuanced relationships between individual data samples.
• Current approaches typically rely on random sample selection, potentially missing crucial local structural information.

Purpose of the Study:

• To introduce Neighborhood Relation-Based Knowledge Distillation (NRKD) as an advanced method for knowledge transfer in model compression.
• To leverage the local structure and inter-sample relationships within data for more effective distillation.
• To enhance the performance of student models by incorporating novel relational knowledge.

Main Methods:

• NRKD identifies K-nearest neighbors for selected samples based on a mini-batch similarity matrix.
• It constructs neighborhood relationship knowledge, capturing local data structures.
• This relational knowledge is transferred via intermediate feature maps and output logits.

Main Results:

• Extensive experiments were conducted on CIFAR10, CIFAR100, Tiny ImageNet, and ImageNet datasets.
• NRKD demonstrated competitive performance compared to state-of-the-art knowledge distillation methods.
• The proposed method effectively transfers relational knowledge for improved student model accuracy.

Conclusions:

• Neighborhood Relation-Based Knowledge Distillation (NRKD) offers a novel and effective approach to model compression.
• Considering local sample structures significantly enhances knowledge transfer in distillation.
• NRKD provides a valuable advancement in the field of efficient deep learning models.