Zero-Shot Evolutionary Architecture Search for Low-Rank Adaptation.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Fine-tuning large Transformer models is computationally intensive due to vast parameter spaces.
• Low-Rank Adaptation (LoRA) offers a parameter-efficient alternative but optimal configuration selection is difficult.

Purpose of the Study:

• To develop a method for efficiently evaluating and discovering optimal LoRA configurations.
• To reduce the computational cost associated with fine-tuning large models.

Main Methods:

• Propose Gradient Projection Score (GPS), a zero-shot proxy metric for rapid LoRA configuration evaluation.
• Introduce EvoLoRA, a zero-shot evolutionary architecture search method optimizing performance, stability, and parameter size.
• Validate GPS correlation with final model performance and EvoLoRA's effectiveness across models and datasets.

Main Results:

• GPS demonstrates strong correlation with final model performance.
• EvoLoRA automatically discovers effective LoRA configurations for diverse tasks.
• EvoLoRA significantly reduces search and training costs for image classification and object detection.

Conclusions:

• GPS provides an efficient proxy for evaluating LoRA configurations.
• EvoLoRA offers an automated and effective approach to optimize LoRA architecture search.
• The proposed methods enhance the efficiency and performance of fine-tuning large Transformer models.