Boosting the Performance of Decentralized Federated Learning via Catalyst Acceleration

Area of Science:

• Machine Learning
• Artificial Intelligence
• Distributed Systems

Background:

• Decentralized Federated Learning (DFL) offers privacy and efficiency but suffers from data heterogeneity, leading to slow convergence and poor generalization.
• Centralized Federated Learning relies on server aggregation, while DFL uses peer-to-peer client connections.

Purpose of the Study:

• To propose an accelerated Decentralized Federated Learning algorithm, DFedCata, that effectively addresses data heterogeneity.
• To improve convergence speed, reduce computational costs, and enhance generalization performance in DFL.

Main Methods:

• Introduction of Catalyst Acceleration, incorporating the Moreau envelope function to mitigate parameter inconsistencies caused by data heterogeneity.
• Integration of Nesterov's extrapolation step to accelerate the model aggregation phase in DFL.
• Theoretical analysis including optimization and generalization error bounds.

Main Results:

• DFedCata demonstrates significant improvements in convergence speed and generalization performance on CIFAR10/100 and Tiny-ImageNet datasets.
• The algorithm shows reduced computational costs compared to existing DFL methods.
• Empirical results strongly align with theoretical predictions, validating the algorithm's effectiveness.

Conclusions:

• DFedCata effectively overcomes the challenges of data heterogeneity in Decentralized Federated Learning.
• The proposed algorithm offers a promising solution for faster, more accurate, and efficient decentralized machine learning.
• The study provides theoretical insights into hyperparameter selection for DFL algorithms.