Federated Fine-Tuning on Heterogeneous LoRAs With Error-Compensated Aggregation

Area of Science:

• Artificial Intelligence
• Machine Learning
• Natural Language Processing

Background:

• Federated learning (FL) enables collaborative training of machine learning models without centralizing data.
• Parameter-efficient fine-tuning (PEFT) methods like low-rank adaptation (LoRA) are crucial for adapting large language models (LLMs) with limited resources.
• Client resource heterogeneity in FL leads to the 'bucket effect,' limiting model performance by the least capable client.

Purpose of the Study:

• To address the limitations of existing LoRA aggregation methods in federated learning for heterogeneous clients.
• To propose a novel method, ECLoRA, that enhances the efficiency and accuracy of federated fine-tuning with heterogeneous LoRA ranks.
• To improve the practicality of federated fine-tuning for LLMs by reducing aggregation overhead and improving convergence speed.

Main Methods:

• Developed ECLoRA, a federated fine-tuning approach utilizing heterogeneous LoRA ranks across clients.
• Employed randomized singular value decomposition (RSVD) to significantly decrease the computational overhead of LoRA aggregation.
• Introduced an error compensation (EC) mechanism to mitigate precision loss by incorporating previous decomposition errors.

Main Results:

• ECLoRA demonstrated significant improvements in final model performance across four foundation models and six public tasks.
• The method achieved accelerated convergence, with average speedups ranging from 1.54x to 3.01x.
• Aggregation time was reduced by approximately 40x compared to classical SVD, highlighting practical efficiency gains.

Conclusions:

• ECLoRA effectively handles client resource heterogeneity in federated LLM fine-tuning.
• The proposed method offers a practical, accurate, and fast solution for federated parameter-efficient fine-tuning.
• ECLoRA represents a significant advancement in making federated LLM adaptation more scalable and efficient.