LoRASculpt: Harmonious Low-Rank Adaptation for Multimodal Large Language Models.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Natural Language Processing

Background:

• Multimodal Large Language Models (MLLMs) possess strong generalization capabilities but face challenges in adapting to downstream tasks without losing general knowledge.
• Low-Rank Adaptation (LoRA) offers parameter-efficient fine-tuning but can introduce redundant updates, leading to forgetting and hindering adaptation.

Purpose of the Study:

• To introduce LoRASculpt+, a novel framework for harmonizing general and specialized knowledge in MLLMs.
• To address the limitations of standard LoRA in preventing redundant updates and catastrophic forgetting during downstream adaptation.

Main Methods:

• Proposed sparse LoRA updates with theoretical guarantees for precise and compact knowledge injection.
• Developed an asymmetric adaptation strategy considering the structural heterogeneity of MLLMs (LLM and connector modules).
• Introduced regularization terms to optimize LoRA updates, steering away from critical pretrained regions and enhancing subspace expressiveness.

Main Results:

• LoRASculpt+ demonstrated enhanced generalization and downstream performance across diverse tasks, model scales, and backbones, even with high sparsity.
• The method effectively enables efficient and precise knowledge injection into MLLMs.
• Catastrophic forgetting was significantly mitigated, facilitating harmonious downstream adaptation.

Conclusions:

• LoRASculpt+ provides an effective solution for parameter-efficient adaptation of MLLMs.
• The framework successfully balances the retention of general knowledge with the acquisition of specialized task-specific knowledge.
• This work advances the field of multimodal learning by enabling more robust and efficient adaptation of large-scale models.