Toward Understanding Generalization and Stability Gaps Between Centralized and Decentralized Federated Learning.

Area of Science:

• Machine Learning
• Distributed Systems

Background:

• Federated learning (FL) encompasses both centralized (CFL) and decentralized (DFL) frameworks, widely applied in practice.
• Current research lacks definitive comparisons of CFL and DFL performance, particularly regarding generalization and stability.
• DFL shows promise in convergence with reduced communication but often underperforms empirically.

Purpose of the Study:

• To comprehensively compare the efficiency and generalization capabilities of CFL and DFL.
• To provide theoretical and empirical evidence for selecting appropriate FL frameworks.
• To identify key factors influencing the performance of both CFL and DFL.

Main Methods:

• Theoretical analysis of stability and generalization on smooth non-convex objectives.
• Mathematical proofs for generalization bounds of CFL and DFL.
• Extensive experimental validation across common FL setups and scenarios.

Main Results:

• CFL consistently generalizes better than DFL.
• Partial participation in CFL yields optimal performance compared to full participation.
• DFL necessitates specific network topologies to maintain performance with increasing training scale.

Conclusions:

• CFL offers superior generalization performance in federated learning.
• Framework selection in FL should consider participation strategies and network topology.
• The findings offer practical guidance for optimizing federated learning deployments.