AdaDFKD: Exploring adaptive inter-sample relationship in data-free knowledge distillation

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Data-free knowledge distillation (DFKD) enables model training without direct data access, crucial for privacy and large-scale transmission.
• Existing DFKD methods struggle with static distributions and teacher model dependency.
• Instance-level distribution learning limits adaptability in prior DFKD approaches.

Purpose of the Study:

• Introduce AdaDFKD, a novel DFKD approach.
• Address limitations of static distributions and teacher model reliance in DFKD.
• Develop an adaptive DFKD method that utilizes relationships among pseudo samples.

Main Methods:

• Generate pseudo samples adaptively from easy-to-hard.
• Employ a relationship refinement module (R2M) for optimizing pseudo-sample generation.
• Learn a progressive conditional distribution of negative samples and maximize inter-sample similarity.

Main Results:

• AdaDFKD demonstrates superiority over state-of-the-art DFKD methods.
• Achieved strong performance across diverse benchmarks and model pairs.
• Exhibited robustness and fast convergence properties.

Conclusions:

• AdaDFKD effectively mitigates risks associated with traditional DFKD.
• The proposed method enhances knowledge distillation by learning adaptive pseudo-sample relationships.
• AdaDFKD offers a more robust and efficient solution for data-free knowledge distillation.