Data-free knowledge distillation via text-noise fusion and dynamic adversarial temperature

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Data-Free Knowledge Distillation (DFKD) enables knowledge transfer without original data.
• Existing DFKD methods struggle with low-quality samples due to noise lacking semantic information.
• Fixed temperatures in adversarial training limit synthesized data diversity.

Purpose of the Study:

• To propose a novel DFKD method, Text-Noise Fusion and Dynamic Adversarial Temperature (TNFDAT).
• To enhance sample quality and diversity in DFKD.
• To improve the generalization and robustness of student networks.

Main Methods:

• Combines random noise with class-specific text embeddings (CSTE) for sample generation.
• Employs dynamic adjustment of adversarial training temperature for the generator.
• Introduces an adaptive sample weighting strategy based on information entropy.

Main Results:

• TNFDAT generates higher-quality training samples by leveraging CSTE.
• Dynamic temperature adjustment significantly improves synthesized sample diversity.
• Adaptive sample weighting enhances knowledge distillation effectiveness and student network robustness.

Conclusions:

• TNFDAT outperforms state-of-the-art DFKD methods.
• The proposed method effectively addresses limitations in sample quality and diversity.
• TNFDAT offers a robust framework for knowledge distillation without requiring original data.