LoLDU: Low-Rank Adaptation via Lower-Diag-Upper Decomposition for Parameter-Efficient Fine-Tuning

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Large model fine-tuning demands significant computational resources.
• Existing methods like Low-Rank Adaptation (LoRA) reduce trainable parameters but can lead to suboptimal convergence and accuracy gaps.
• Approximating weight updates with random initialization in LoRA can hinder performance compared to full fine-tuning.

Purpose of the Study:

• Introduce Low-rank LDU (LoLDU), a novel parameter-efficient fine-tuning (PEFT) approach.
• Address the limitations of existing PEFT methods, including suboptimal convergence and accuracy gaps.
• Significantly reduce the number of trainable parameters while maintaining model performance.

Main Methods:

• Utilize lower-diag-upper (LDU) decomposition for initializing low-rank matrices.
• Employ LDU decomposition to ensure faster convergence and nonsingularity of matrices.
• Focus on optimizing the diagonal matrix for scaling transformations, minimizing trainable parameters.

Main Results:

• LoLDU reduces trainable parameters by 2600 times compared to regular PEFT methods.
• Achieves performance comparable to full fine-tuning across various tasks and models.
• Demonstrates the fewest parameters among all known PEFT approaches.

Conclusions:

• LoLDU offers a highly efficient PEFT method with minimal trainable parameters.
• The LDU decomposition provides a robust initialization strategy for improved fine-tuning.
• LoLDU presents a promising solution for resource-constrained fine-tuning of large-scale models.