Making Knowledge Distillation Open Again

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Knowledge distillation (KD) transfers knowledge from large teacher models to smaller student models.
• Traditional feature-based KD risks data privacy and proprietary information leakage.
• These risks limit KD deployment in collaborative or cloud environments.

Purpose of the Study:

• To propose a privacy-aware, teacher-oriented projected feature distillation framework.
• To reduce direct exposure of teacher model's intermediate representations.
• To maintain distillation effectiveness while enhancing privacy.

Main Methods:

• Employs a low-rank projection strategy to obfuscate teacher features into a compact subspace.
• Student models align their features using a projection matrix derived from the teacher.
• Enables knowledge transfer while minimizing direct exposure of original spatial patterns.

Main Results:

• Demonstrates empirical feature obfuscation and resistance to reconstruction.
• Achieves competitive performance in object detection and semantic segmentation.
• Shows performance gains even when learning from projected representations.

Conclusions:

• The proposed framework effectively balances knowledge transfer and privacy preservation.
• Projected feature distillation offers a viable solution for privacy-sensitive KD applications.
• This method facilitates broader adoption of KD in secure computing scenarios.