La-LoRA: Parameter-efficient fine-tuning with layer-wise adaptive low-rank adaptation.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Parameter-efficient fine-tuning (PEFT) adapts large models efficiently.
• Low-Rank Adaptation (LoRA) is a popular PEFT method, freezing pre-trained weights and using low-rank matrices.
• LoRA's uniform rank assignment across layers is suboptimal due to varying layer importance.

Purpose of the Study:

• To introduce Layer-wise Adaptive Low-Rank Adaptation (La-LoRA) for improved PEFT.
• To address LoRA's limitation of uniform rank allocation.
• To enhance model adaptation by dynamically allocating ranks based on layer contribution.

Main Methods:

• Proposed La-LoRA, a novel PEFT approach.
• Implemented Dynamic Contribution-Driven Parameter Budget (DCDPB) for rank allocation.
• Utilized Truncated Norm Weighted Dynamic Rank Allocation (TNW-DRA) for progressive rank adjustment.
• Treated each layer as an independent unit for rank optimization.

Main Results:

• La-LoRA demonstrated consistent performance improvements across various tasks and models.
• The proposed method outperformed existing PEFT benchmarks.
• Dynamic rank allocation proved effective in optimizing model adaptation.

Conclusions:

• La-LoRA offers superior performance and computational efficiency compared to standard LoRA.
• The layer-wise adaptive approach effectively handles the heterogeneous importance of model layers.
• La-LoRA provides a flexible and effective solution for adapting large pre-trained models.