Selective knowledge sharing for privacy-preserving federated distillation without a good teacher

Area of Science:

• Artificial Intelligence
• Machine Learning
• Distributed Systems

Background:

• Federated learning (FL) offers privacy-preserving collaborative training but faces challenges like privacy attacks, communication overhead, and model heterogeneity.
• Federated distillation (FD) addresses FL limitations by transferring knowledge instead of model parameters, but suffers from ambiguous knowledge sharing due to data variations and lack of a teacher model.

Purpose of the Study:

• To propose a novel selective knowledge sharing mechanism for federated distillation (FD) to improve its performance and robustness.
• To enhance the privacy-preserving, communication-efficient, and heterogeneity-adaptive capabilities of federated training frameworks.

Main Methods:

• Introduced Selective-FD, a mechanism for identifying accurate and precise knowledge from local and ensemble predictions within the FD framework.
• Conducted empirical studies and provided theoretical insights to validate the proposed approach.

Main Results:

• Selective-FD significantly enhances the generalization capabilities of the federated distillation framework.
• The proposed approach consistently outperforms existing baseline methods in federated learning scenarios.

Conclusions:

• Selective-FD effectively addresses the challenges of ambiguous knowledge sharing in FD, leading to improved model performance.
• The study paves the way for more robust and efficient federated training systems adaptable to diverse data distributions and model architectures.