Efficient sparse low-rank adaptation for large language models

Parameter-efficient fine-tuning (PEFT) methods adapt foundation models to specific domains by selectively updating only a small subset of critical parameters, significantly reducing computational costs. Among these methods, Sparse Low-Rank Adaptation (SoRA) has emerged as an effective approach for fine-tuning large language models. This paper investigates SoRA algorithm's performance in large-scale models, particularly analyzing learning dynamics and scaling effects in wide neural networks. We find that employing a uniform learning rate across all parameter matrices limits learning efficiency. To address this, we derive an improved learning rate configuration and propose SoRA+, an enhanced variant that introduces a hyperparameter to adjust learning rates proportionally. This modification aligns with theoretical optima and mitigates the suboptimality of the original SoRA method. Experiments show that SoRA+ improves fine-tuning performance by up to 2% while accelerating convergence.