GSLoRa: Gradient spectral alignment for low-rank adaptation

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Machine Learning

Background:

• Parameter-efficient fine-tuning (PEFT) adapts large language models (LLMs) with minimal updates.
• Existing PEFT methods face challenges like slow convergence, gradient noise, and poor feature alignment.

Purpose of the Study:

• To develop an advanced low-rank adaptation framework for PEFT.
• To address the limitations of current PEFT techniques for improved efficiency and stability.

Main Methods:

• Developed a low-rank adaptation framework utilizing gradient spectral alignment.
• Implemented spectral decomposition with eigenvalue scaling (retaining 95% spectral energy).
• Designed a dual gradient projector (orthogonal and spherical) to reduce gradient noise variance by 38.2% compared to LoRA.
• Introduced dynamic eigenvalue scaling for adaptive recalibration of principal component weights.

Main Results:

• Achieved performance comparable to full fine-tuning with only 0.1% parameters trained.
• Outperformed AdaLoRA by 6.2% and DoRA by 5.3% on the GLUE benchmark.
• Improved mathematical reasoning accuracy of LLaMA-7B on GSM8K by 7.8%.

Conclusions:

• The proposed GSLoRA framework offers significant improvements in PEFT efficiency, stability, and adaptability.
• Novel insights into gradient spectral alignment enhance LLM adaptation capabilities.
• The method demonstrates superior performance and adaptability across various NLP tasks.