Fed-HeLLo: Efficient Federated Foundation Model Fine-Tuning With Heterogeneous LoRA Allocation.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Distributed Systems

Background:

• Federated learning (FL) enables collaborative fine-tuning of foundation models (FMs) across clients.
• Federated low-rank adaptation (LoRA) methods allow efficient local fine-tuning with fewer parameters.
• Existing methods often overlook client resource heterogeneity and optimal local training strategies.

Purpose of the Study:

• To propose Fed-HeLLo, a federated LoRA-based fine-tuning framework addressing client resource heterogeneity.
• To develop adaptive heterogeneous LoRA allocation (HLA) strategies for optimizing global fine-tuning performance.

Main Methods:

• Fed-HeLLo enables clients to fine-tune FMs with varying local trainable LoRA layers.
• Developed HLA strategies include Fisher Information Matrix (FIM-HLA) based on dynamic layer importance and geometrically defined (GD-HLA) strategies for stability.
• Introduced randomized GD-HLA (RGD-HLA) for enhanced accuracy.

Main Results:

• Fed-HeLLo effectively allocates LoRA layers based on client resources and layer importance.
• FIM-HLA and GD-HLA strategies demonstrated significant improvements in federated fine-tuning.
• Evaluations across diverse datasets and non-i.i.d. distributions confirmed the framework's effectiveness and efficiency.

Conclusions:

• Fed-HeLLo provides an effective solution for heterogeneous federated LoRA-based fine-tuning.
• The proposed HLA strategies enhance performance, stability, and accuracy in resource-constrained FL settings.
• This work offers a practical framework for collaborative FM fine-tuning in diverse environments.