Understanding the Effects of Projectors in Knowledge Distillation

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Knowledge distillation typically uses projectors for feature transformation due to dimension mismatches.
• The role of projectors in knowledge distillation, even without dimension mismatch, is not well understood.

Purpose of the Study:

• Investigate the overlooked implicit role of projectors in knowledge distillation.
• Propose a novel projector ensemble-based feature distillation method.

Main Methods:

• Empirical study analyzing projector effects on student network accuracy, feature similarity (using Centered Kernel Alignment - CKA), and over-confidence.
• Development and evaluation of a projector ensemble-based feature distillation strategy.

Main Results:

• Projectors improve the accuracy trade-off and preserve teacher similarity beyond numeric resemblance.
• Projectors help prevent student models from becoming over-confident.
• The proposed projector ensemble method demonstrates superior classification performance and improved feature quality (verified by CKA and model calibration).

Conclusions:

• Projectors play a crucial, often overlooked, role in enhancing knowledge distillation performance and student model characteristics.
• The projector ensemble approach offers a simple yet effective strategy for improving feature distillation and model calibration.