E-3SFC: Communication-Efficient Federated Learning With Double-Way Features Synthesizing.

Area of Science:

• Machine Learning
• Distributed Systems
• Optimization

Background:

• Federated learning (FL) faces communication burdens due to growing model sizes.
• Current gradient compression methods in FL suffer from high compression errors, hindering convergence.
• Efficient communication is crucial for practical FL deployment.

Purpose of the Study:

• To develop a novel gradient compression algorithm for federated learning that achieves high compression effectiveness and low error.
• To reduce communication overhead in FL without compromising model convergence and accuracy.
• To provide theoretical guarantees and empirical validation for the proposed method.

Main Methods:

• Propose extended single-step synthetic features compressing (E-3SFC), comprising single-step synthetic features compressor (3SFC), double-way compression (DWC), and communication budget scheduler (BS).
• Utilize the model itself as a decompressor to compress raw gradients into synthetic features, incorporating error feedback (EF).
• Conduct theoretical analysis for strongly convex and non-convex settings, and perform extensive experiments on six datasets and six models.

Main Results:

• E-3SFC significantly reduces communication costs by up to 111.6 times.
• The proposed method outperforms state-of-the-art techniques by up to 13.4% in performance.
• Theoretical analysis confirms linear and sublinear convergence rates under aggregation noise.

Conclusions:

• E-3SFC effectively addresses the communication bottleneck in federated learning.
• The novel approach enhances communication efficiency without negatively impacting model performance.
• This work offers a promising direction for efficient and effective federated learning systems.