Communication-efficient federated learning via knowledge distillation

Area of Science:

• Machine Learning
• Artificial Intelligence
• Data Privacy

Background:

• Federated learning trains models on decentralized data, preserving privacy by transmitting model updates instead of raw data.
• Large model updates and numerous communication rounds in federated learning incur substantial communication costs, leading to client overhead and environmental concerns.

Purpose of the Study:

• To introduce FedKD, a novel federated learning method designed for enhanced communication efficiency and effectiveness.
• To address the significant communication overheads associated with traditional federated learning approaches.

Main Methods:

• FedKD employs adaptive mutual knowledge distillation to improve model learning.
• Dynamic gradient compression techniques are integrated to reduce the size of model updates.
• The method was validated across three distinct privacy-sensitive scenarios.

Main Results:

• FedKD maximally reduced communication costs by 94.89% in tested scenarios.
• The method achieved competitive performance compared to centralized model learning.
• Effectiveness demonstrated across multiple privacy-preserving applications.

Conclusions:

• FedKD offers a communication-efficient and effective solution for federated learning.
• The approach has the potential for widespread deployment in privacy-preserving intelligent systems.
• Applications include intelligent healthcare and personalized services.