Optimizing Client Participation in Communication-Constrained Federated LLM Adaptation with LoRA

Area of Science:

• Artificial Intelligence
• Machine Learning
• Distributed Systems

Background:

• Federated learning (FL) facilitates privacy-preserving adaptation of large language models (LLMs).
• High communication overhead in FL hinders deployment in edge environments.
• Parameter-efficient fine-tuning (PEFT), like low-rank adaptation (LoRA), reduces LLM update sizes.

Purpose of the Study:

• To propose LoRaC-GA, a communication-aware optimization framework for FL with LLMs.
• To dynamically determine the optimal number of clients per round under bandwidth constraints.
• To maximize both model accuracy and communication efficiency.

Main Methods:

• Formulated a max-min objective for joint accuracy and communication efficiency.
• Employed a genetic algorithm (GA) to solve the non-convex optimization problem.
• Integrated a structured peer-to-peer collaboration protocol with log2K complexity.

Main Results:

• LoRaC-GA adaptively selects the optimal client count for each round.
• The framework achieves competitive accuracy with significantly reduced communication costs.
• Demonstrated effectiveness in bandwidth-constrained edge deployments.

Conclusions:

• LoRaC-GA enhances the feasibility of FL for large-scale LLMs in edge settings.
• The framework offers a scalable and efficient solution for communication-limited environments.
• Optimized client selection is crucial for efficient federated LLM adaptation.