Mixture of Cluster-Conditional LoRA Experts for Vision-Language Instruction Tuning

Area of Science:

• Artificial Intelligence
• Computer Vision
• Natural Language Processing

Background:

• Instruction tuning has enabled Large Vision-language Models (LVLMs) to achieve zero-shot generalization across diverse vision-language tasks.
• Training LVLMs on varied data sources can cause task conflicts, degrading instruction-following performance due to parameter competition.

Purpose of the Study:

• To propose a novel architecture that mitigates task conflicts in instruction-tuned LVLMs.
• To enhance the instruction-following capabilities and generalization of LVLMs.

Main Methods:

• Introduced Mixture of Cluster-conditional LoRA Experts (MoCLE), a Mixture of Experts (MoE) architecture.
• MoCLE activates task-specific LoRA (Low-Rank Adaptation) parameters based on clustered instructions.
• Incorporated a universal expert to improve generalization to unseen instructions.

Main Results:

• MoCLE effectively addresses task conflicts by selectively activating expert parameters.
• Experiments on InstructBLIP and LLaVA datasets validate the proposed MoCLE architecture's effectiveness.
• The model demonstrates improved performance in instruction following and generalization.

Conclusions:

• MoCLE offers a promising solution for optimizing LVLM performance by managing diverse training tasks.
• The proposed architecture enhances the versatility and robustness of LVLMs for various downstream applications.