Communication-Efficient Nonconvex Federated Learning With Error Feedback for Uplink and Downlink.

Area of Science:

• Machine Learning
• Distributed Systems
• Optimization

Background:

• Large-scale online learning often involves nonconvex distributed optimization, which is computationally challenging.
• Federated learning systems face communication bottlenecks, particularly with limited uplink bandwidth from edge devices.

Purpose of the Study:

• To develop communication-efficient nonconvex federated learning algorithms.
• To address the challenges of uplink and downlink communication asymmetry in federated learning.

Main Methods:

• Proposed two novel algorithms: Error Feedback 2021 (EF21) and Lazily Aggregated Gradient (LAG).
• Developed EF21 with LAG (EF-LAG) to reduce uplink communication costs.
• Introduced Bidirectional EF-LAG (BiEF-LAG) for reducing both uplink and downlink costs.

Main Results:

• EF21 demonstrates improved performance over vanilla Error Feedback.
• EF-LAG and BiEF-LAG significantly reduce communication overhead.
• The proposed algorithms achieve convergence rates comparable to Gradient Descent (GD) for smooth nonconvex functions.

Conclusions:

• The developed algorithms effectively reduce communication costs in nonconvex federated learning.
• These methods maintain learning quality while enhancing communication efficiency.
• Empirical results confirm the superiority of the proposed algorithms on synthetic and deep learning benchmarks.