MoFedAGR: Mitigating client drift with adaptive gradient regularization and global momentum in federated learning

Area of Science:

• Machine Learning
• Distributed Systems
• Data Science

Background:

• Federated learning (FL) offers privacy-preserving distributed machine learning but struggles with heterogeneous data.
• Heterogeneous data leads to client drift, where local models overfit and deviate from the global optimum.
• Client drift degrades model performance and generalization in federated learning settings.

Purpose of the Study:

• To address the performance degradation and generalization issues caused by client drift in federated learning.
• To develop a novel method that mitigates aggregation error stemming from heterogeneous data.
• To improve the convergence of client models to a consistent flat minimum.

Main Methods:

• Proposed adaptive gradient regularization (AGR) that applies varying regularization strengths based on parameter variance between local and global models.
• Introduced global momentum from the server as a gradient correction term to approximate global gradients.
• Developed MoFedAGR, integrating gradient correction and adaptive gradient regularization for enhanced client model convergence.
• Provided theoretical convergence bounds for the proposed algorithm.

Main Results:

• MoFedAGR significantly improves model performance on image classification tasks.
• The proposed method demonstrates strong generalization capabilities.
• Experiments validate the effectiveness of adaptive gradient regularization and global momentum in mitigating client drift.

Conclusions:

• MoFedAGR effectively addresses client drift in federated learning by promoting convergence to flatter minima.
• The approach enhances both performance and generalization in privacy-preserving distributed machine learning.
• This work offers a robust solution for federated learning with heterogeneous data.