Robust and Communication-Efficient Federated Learning From Non-i.i.d. Data.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Data Privacy

Background:

• Federated learning (FL) facilitates collaborative deep learning model training across multiple parties without centralizing sensitive data.
• A major challenge in FL is the substantial communication overhead during the training process.
• Existing compression techniques for distributed training offer limited utility in FL due to issues with downstream communication and data distribution assumptions.

Purpose of the Study:

• To propose a novel compression framework, Sparse Ternary Compression (STC), specifically designed for the federated learning environment.
• To address the limitations of existing methods by enabling both upstream and downstream communication compression.
• To improve the efficiency of federated learning in bandwidth-constrained settings.

Main Methods:

• Developed Sparse Ternary Compression (STC), extending top-k gradient sparsification.
• Incorporated a novel mechanism for downstream compression.
• Implemented ternarization and optimal Golomb encoding for weight updates.

Main Results:

• STC demonstrated superior performance compared to federated averaging across four distinct learning tasks.
• The proposed method effectively reduces communication overhead in common federated learning scenarios.
• Experiments validated STC's efficacy under non-idealized, heterogeneous data distributions.

Conclusions:

• STC offers a significant advancement in federated learning efficiency by enabling high-frequency, low-bitwidth communication.
• The findings suggest a paradigm shift towards more efficient communication strategies in federated optimization.
• STC is particularly beneficial for bandwidth-constrained federated learning environments.