Data-free knowledge distillation via generator-free data generation for Non-IID federated learning

Area of Science:

• Artificial Intelligence
• Machine Learning
• Distributed Systems

Background:

• Non-IID data heterogeneity is a major challenge in Federated Learning (FL), causing local model drift and performance degradation.
• Existing knowledge distillation methods for FL often rely on proxy datasets or data generators, which are not always available or reliable.
• The instability of generated data and server-dependent generators limit the effectiveness of current approaches.

Purpose of the Study:

• To propose a novel data-free knowledge distillation approach for Non-IID FL, named FedF^2DG.
• To overcome the limitations of proxy datasets and data generators in FL scenarios.
• To enhance the performance of global models in heterogeneous federated environments.

Main Methods:

• FedF^2DG utilizes local models to generate pseudo datasets for each client, enabling data-free knowledge distillation.
• A regularization term is introduced to generate hard samples by exploiting disagreements between local and global models.
• A Data Generation Principle adaptively controls pseudo-dataset label distribution and quantity based on client state.

Main Results:

• FedF^2DG significantly outperforms state-of-the-art FL methods in Non-IID settings.
• The method effectively extracts more client knowledge through adaptive pseudo-dataset generation.
• Experiments show improved performance when FedF^2DG is used as a plugin for existing FL algorithms like FedAvg and FedProx.

Conclusions:

• FedF^2DG offers an effective data-free solution for Non-IID Federated Learning.
• The proposed approach enhances global model accuracy and robustness in heterogeneous data distributions.
• FedF^2DG provides a flexible and performant method for improving FL systems.