The Risk of Federated Learning to Skew Fine-Tuning Features and Underperform Robustness.

Area of Science:

• Machine Learning
• Artificial Intelligence
• Data Science

Background:

• Federated learning (FL) combined with fine-tuning (FT) addresses data scarcity and privacy concerns in domain-specific datasets.
• However, FL can negatively impact the out-of-distribution (OOD) robustness of pretrained models by skewing FT features.

Purpose of the Study:

• To investigate the impact of federated learning on model robustness.
• To propose a novel algorithm to mitigate these negative effects and enhance model robustness.

Main Methods:

• Introduced three robustness indicators to analyze data representations, transferability, and model deviations.
• Developed a general noisy projection (GNP)-based robust algorithm.
• Incorporated transfer of robustness from pretrained to fine-tuned models and added Gaussian noise.

Main Results:

• Federated learning was found to risk skewing FT features and compromising OOD robustness.
• The proposed GNP algorithm effectively enhances model robustness across diverse scenarios.
• The approach maintains accuracy on the target distribution while improving robustness against label and quantity distribution skew.

Conclusions:

• Federated learning presents challenges to model robustness, particularly OOD robustness.
• The GNP algorithm offers a viable solution to enhance robustness in federated fine-tuning without performance degradation.
• This method supports various parameter-efficient FT techniques and different data distribution skews.