Federated Learning Based on Model Discrepancy and Variance Reduction.

Area of Science:

• Machine Learning
• Distributed Systems
• Optimization

Background:

• Data heterogeneity and asynchronous client participation in federated learning (FL) cause model discrepancies and slow global convergence.
• Existing FL methods often struggle with unstable convergence due to client-side variance.

Purpose of the Study:

• To propose novel frameworks, FedVR and FedMDVR, to mitigate variance and enhance convergence in federated learning.
• To address the challenges posed by data heterogeneity and asynchronous client updates in FL.

Main Methods:

• FedVR aggregates fresh and stale client updates at the server to form a control variate, reducing client variance without extra communication.
• FedMDVR further broadcasts this control variate to active clients, guiding their local updates towards the global optimum.
• Theoretical convergence proofs for FedVR and FedMDVR in general non-convex settings are provided.

Main Results:

• FedVR and FedMDVR significantly accelerate convergence by reducing the number of communication rounds needed for target accuracy.
• Both methods demonstrate improved convergence to higher accuracies compared to baseline algorithms.
• Experimental evaluations on benchmark datasets validate the effectiveness of the proposed frameworks.

Conclusions:

• FedVR and FedMDVR offer effective solutions for variance reduction in federated learning.
• The proposed methods enhance both the speed and final accuracy of federated model training.
• These frameworks show promise for improving the stability and performance of distributed machine learning systems.