Decentralized Federated Learning by Partial Message Exchange

Area of Science:

• Artificial Intelligence
• Machine Learning
• Distributed Systems

Background:

• Decentralized federated learning (DFL) enables collaborative machine learning without a central server.
• DFL faces challenges like data heterogeneity, strict theoretical assumptions, and performance degradation with privacy techniques.

Purpose of the Study:

• To develop a novel DFL algorithm addressing communication costs and data privacy.
• To improve convergence rates and relax restrictive assumptions in DFL.

Main Methods:

• Introduced PaME (DFL by Partial Message Exchange), an algorithm exchanging only sparse coordinates between nodes.
• Developed a formal reconstruction risk theory for privacy analysis under partial observation.
• Proved convergence in expectation to a stationary point at a linear rate.

Main Results:

• PaME significantly reduces communication overhead.
• The algorithm limits sensitive data exposure, rigorously characterized by reconstruction risk theory.
• PaME demonstrates superior performance over existing decentralized learning algorithms in experiments.
• Convergence is proven under mild assumptions (Lipschitz continuity, doubly stochastic matrix), effectively handling data heterogeneity.

Conclusions:

• PaME offers an effective solution for decentralized federated learning challenges.
• The algorithm provides a practical approach to reduce communication and enhance privacy.
• PaME's theoretical guarantees and experimental results highlight its advantages for large-scale heterogeneous networks.