MLAE: Masked LoRA Experts for Parameter-Efficient Fine-Tuning

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Full fine-tuning of large models requires extensive parameter updates, posing computational challenges.
• Parameter-efficient fine-tuning (PEFT) methods like Low-Rank Adaptation (LoRA) simplify this but can suffer from redundancy and limited effectiveness.
• Enhancing the independence and diversity of low-rank matrices is crucial for improving PEFT performance.

Purpose of the Study:

• To introduce Masked LoRA Experts (MLAE), an innovative visual PEFT method.
• To address the limitations of existing LoRA methods by increasing the diversity and independence of low-rank matrices.
• To improve model performance and generalization through selective activation of learned components.

Main Methods:

• Proposed MLAE, applying masking concepts to visual PEFT.
• Employed a cellular decomposition strategy, treating rank-1 components as experts within LoRA parameterization.
• Introduced a binary mask matrix for selective expert activation, promoting diverse and anisotropic learning via expert-level dropout.

Main Results:

• MLAE demonstrated enhanced performance and fostered more diverse knowledge acquisition.
• A significant decrease in parameter similarity among MLAE components was observed, boosting model quality.
• Achieved new state-of-the-art (SOTA) performance with 78.8% accuracy on VTAB-1k and 90.9% on FGVC benchmarks, surpassing previous SOTA by 0.8% average.

Conclusions:

• MLAE effectively enhances PEFT by promoting diversity and independence in low-rank matrices.
• The method shows strong generalization capabilities across various tasks, including LLM fine-tuning and semantic segmentation.
• MLAE represents a versatile and effective advancement in parameter-efficient fine-tuning techniques.