Unpacking the Gap Box Against Data-Free Knowledge Distillation.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Data-free knowledge distillation (DFKD) trains student models using a teacher model without requiring original training data.
• Existing DFKD methods struggle with generated sample quality due to the gap between teacher (T) and student (S) model probabilities, leading to suboptimal generalization.
• The ideal teacher (T*) for distillation is unknown, making it difficult to assess the 'goodness' of generated samples.

Purpose of the Study:

• To investigate the 'gap box' in DFKD and develop a method for generating high-quality samples.
• To address the limitations of existing DFKD approaches by proposing a novel sample generation strategy.
• To theoretically and empirically validate the proposed method's effectiveness.

Main Methods:

• Proposed Gap-Sensitive Sample Generation (GapSSG) approach analyzing empirical distilled risk.
• Unpacked the gap between T and S into inherent and derived gaps.
• Tracked student model training to capture category distribution and devised a regulatory factor to approximate T*.
• Implemented a sample-balanced strategy during generator training to mitigate overfitting and knowledge gaps.

Main Results:

• Confirmed the existence of an ideal teacher (T*) and theoretically linked gap disturbance to T-T* mismatch.
• Demonstrated that generated samples should maximize benefit to S via T's class probabilities.
• Showcased GapSSG's ability to generate 'good' samples by approximating T* and adapting to S.
• Empirical studies verified GapSSG's superiority over state-of-the-art methods.

Conclusions:

• GapSSG effectively generates beneficial samples for DFKD by analyzing and bridging the gap between teacher and student models.
• The proposed method improves student model generalization in data-free settings.
• GapSSG offers a significant advancement in data-free knowledge distillation techniques.