FedRAD: Heterogeneous Federated Learning via Relational Adaptive Distillation

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Federated Learning (FL) is a distributed machine learning approach preserving data privacy in the Internet of Things (IoT).
• Non-Independent and Identically Distributed (Non-IID) data across IoT devices degrades FL performance, causing slow convergence and reduced accuracy.
• Existing FL methods struggle to mitigate the 'client forgetting' problem in Non-IID settings.

Purpose of the Study:

• To propose FedRAD, a novel Federated Learning method to address Non-IID data challenges.
• To enhance the retention of global knowledge during local training phases in FL.
• To improve convergence speed and classification accuracy in distributed machine learning for IoT.

Main Methods:

• FedRAD employs relational knowledge distillation to mine high-quality global knowledge from a higher-dimensional perspective.
• An entropy-wise adaptive weights module (EWAW) dynamically balances single-sample and relational knowledge distillation losses.
• Local models are trained to better retain global knowledge and prevent forgetting.

Main Results:

• FedRAD demonstrated superior performance over advanced FL methods in experimental evaluations.
• The proposed method achieved significant improvements in convergence speed.
• Enhanced classification accuracy was observed on CIFAR10 and CIFAR100 datasets.

Conclusions:

• FedRAD effectively tackles the Non-IID data problem in Federated Learning.
• The method enhances global knowledge retention, leading to better FL model performance.
• FedRAD offers a promising solution for privacy-preserving machine learning in diverse IoT environments.