Bidirectional Decoupled Distillation for Heterogeneous Federated Learning | JoVE Visualize

Area of Science:

Artificial Intelligence
Machine Learning
Distributed Systems

Background:

Federated learning (FL) enables collaborative model training across decentralized devices while preserving data privacy.
Data heterogeneity across clients in FL can lead to a global model that compromises individual client performance and personalization.
Existing federated distillation methods often struggle with non-target class features, limiting local model convergence.

Purpose of the Study:

To introduce a novel approach, Bidirectional Decoupled Distillation For Heterogeneous Federated Learning (BDD-HFL), to address the client personalization challenge in FL.
To enhance the distillation process by enabling bidirectional knowledge exchange between local and private models within each client.
To improve the convergence and accuracy of local models in heterogeneous federated environments.

Main Methods:

Proposed BDD-HFL, incorporating an auxiliary private model on each client for bidirectional knowledge exchange.
Decomposed network outputs into target and non-target class logits for separate distillation.
Employed a joint optimization strategy using cross-entropy and decoupled relative-entropy loss for enhanced feature learning.
Evaluated BDD-HFL on CIFAR-10, CIFAR-100, and MNIST datasets under IID, Non-IID, and unbalanced data distributions.

Main Results:

BDD-HFL demonstrated superior performance compared to state-of-the-art federated distillation methods across multiple baselines.
Achieved up to a 3% improvement in average classification accuracy on benchmark datasets.
Showcased effectiveness in IID, Non-IID, and unbalanced data distribution scenarios, highlighting robustness.

Conclusions:

BDD-HFL effectively addresses the personalization challenge in heterogeneous federated learning by enabling bidirectional knowledge distillation.
The proposed decoupled distillation strategy enhances the learning of both target and non-target class features, leading to better local model convergence.
BDD-HFL exhibits strong generalization capabilities, offering a promising solution for practical federated learning applications requiring personalized models.