Ternary Compression for Communication-Efficient Federated Learning.

Area of Science:

• Machine Learning
• Distributed Systems
• Data Privacy

Background:

• Federated learning (FL) enables collaborative model training across decentralized devices, crucial for privacy-sensitive applications with mobile and IoT data.
• Current FL methods often use full-precision models, leading to high communication costs due to redundant parameters.
• Existing approaches struggle with the trade-off between communication efficiency and model performance.

Purpose of the Study:

• To develop a novel federated learning algorithm that minimizes communication overhead.
• To address the redundancy in model parameters within federated learning systems.
• To enhance the efficiency of privacy-preserving machine learning on decentralized data.

Main Methods:

• Proposing a federated trained ternary quantization (FTTQ) algorithm for optimizing quantized networks on clients using a self-learning quantization factor.
• Developing a ternary federated averaging (T-FedAvg) protocol built upon FTTQ to reduce communication costs.
• Providing theoretical convergence proofs for quantization factors, unbiasedness, and reduced weight divergence.

Main Results:

• The proposed T-FedAvg effectively reduces both upstream and downstream communication costs in federated learning.
• Empirical experiments demonstrate the algorithm's effectiveness on widely used deep learning models.
• The T-FedAvg protocol achieves comparable or even slightly improved performance on non-IID data compared to standard federated learning algorithms.

Conclusions:

• FTTQ and T-FedAvg offer a significant advancement in communication efficiency for federated learning.
• The proposed methods maintain or enhance model performance, particularly in challenging non-IID data scenarios.
• This work provides a practical solution for large-scale, privacy-preserving machine learning applications.