Generalized Knowledge Distillation via Relationship Matching.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Deep neural networks (teachers) possess valuable knowledge for related tasks.
• Knowledge distillation transfers teacher knowledge to student models, enhancing learning.
• Existing methods often require teachers and students to share the same task or label space.

Purpose of the Study:

• To propose a novel Generalized Knowledge Distillation (GKD) framework that allows for flexible task and label space alignment between teacher and student models.
• To introduce the RElationship FacIlitated Local cLassifiEr Distillation (ReFilled) method that leverages instance comparison abilities for cross-task knowledge transfer.
• To demonstrate the effectiveness of ReFilled across various scenarios, including standard knowledge distillation, incremental learning, and few-shot learning.

Main Methods:

• Generalized Knowledge Distillation (GKD) allows teacher and student models to have different, partially overlapped, or identical label spaces.
• The ReFilled method reweights hard tuples from the student model and aligns similarity comparison levels between instances.
• An embedding-induced classifier from the teacher model guides the student's classification confidence and emphasizes relevant supervision.

Main Results:

• ReFilled exhibits strong discriminative capabilities even when teacher and student classes are completely non-overlapping.
• The method achieves state-of-the-art performance on standard knowledge distillation tasks.
• ReFilled also demonstrates superior results in one-step incremental learning and few-shot learning benchmarks.

Conclusions:

• Instance comparison is a crucial factor for effective knowledge transfer across diverse tasks in deep learning.
• The proposed ReFilled method offers a robust and flexible approach to Generalized Knowledge Distillation.
• ReFilled significantly improves student model performance in various learning paradigms, showcasing its broad applicability.