Data free knowledge distillation with feature synthesis and spatial consistency for image analysis

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Privacy and security concerns limit access to original training data, hindering model compression techniques.
• Data-Free Knowledge Distillation (DFKD) offers a solution by transferring knowledge without raw data access.
• Existing DFKD methods face challenges in generating high-fidelity synthetic data and preserving spatial attributes, leading to suboptimal performance.

Purpose of the Study:

• To propose a novel DFKD strategy that overcomes limitations of existing methods in synthetic data generation and spatial attribute preservation.
• To enhance knowledge transfer from teacher to student networks in a data-free setting.
• To improve the generalization capabilities of compressed models.

Main Methods:

• An enhanced DCGAN generator with an attention module was developed for synthesizing high-quality samples with improved micro-discriminative features.
• A Multi-Scale Spatial Activation Region Consistency (MSARC) mechanism was introduced to accurately replicate the teacher network's spatial attributes.
• An adversarial learning framework was employed to create a dynamic competitive environment between generative and distillation processes.

Main Results:

• The proposed DFKD method demonstrated superior performance across benchmark datasets including CIFAR-10, CIFAR-100, Tiny-ImageNet, PathMNIST, BloodMNIST, and PneumoniaMNIST.
• On CIFAR-100, the student network achieved 77.85% accuracy, outperforming prior methods like CMI and SpaceshipNet.
• On BloodMNIST, the method attained 80.50% accuracy, exceeding the next best method by over 5%.

Conclusions:

• The novel DFKD strategy effectively addresses challenges in data-free knowledge transfer by improving synthetic data quality and preserving spatial information.
• The method shows significant potential for privacy-preserving model compression, particularly in domains with sensitive data like medical imaging.
• The proposed approach offers a robust and efficient solution for knowledge distillation in data-constrained environments.