CKD: Contrastive Knowledge Distillation From a Sample-Wise Perspective.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Conventional knowledge distillation relies heavily on feature similarity, risking overfitting.
• Contrastive methods often prioritize inter-class discrimination over intra-sample semantic relationships.

Purpose of the Study:

• To propose a contrastive knowledge distillation framework for sample-wise logit alignment and semantic consistency.
• To address limitations of existing knowledge distillation and contrastive learning approaches.

Main Methods:

• Implemented a teacher-student contrastive alignment framework at the sample level.
• Enforced intra-sample alignment by minimizing logit discrepancies and inter-sample contrasts for semantic dissimilarities.
• Utilized an InfoNCE loss framework, reducing computational complexity and eliminating dependencies on temperature parameters and large batch sizes.

Main Results:

• Demonstrated effectiveness on image classification, object detection, and instance segmentation tasks.
• Achieved superior performance compared to conventional methods through comprehensive experiments on CIFAR-100, ImageNet-1K, and MS COCO datasets.

Conclusions:

• The proposed contrastive knowledge distillation framework effectively transfers "dark knowledge" by aligning sample-wise logits and preserving semantic consistency.
• The method offers a computationally efficient and robust approach for various computer vision applications.