MoIL: Momentum Imitation Learning for Efficient Vision-Language Adaptation.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Large-scale vision-language pre-training (VLP) models are crucial but computationally expensive to fine-tune.
• Existing parameter-efficient fine-tuning (PEFT) methods like LoRA have limitations in VLP, including approximation errors and inefficient optimization.
• High storage costs associated with full fine-tuning hinder the widespread adoption of VLP models.

Purpose of the Study:

• To develop a novel PEFT method for VLP models that overcomes the limitations of existing approaches.
• To improve the approximation accuracy and optimization efficiency of low-rank adaptation in VLP.
• To reduce the computational and storage costs associated with adapting large VLP models.

Main Methods:

• Propose Momentum Imitation Learning (MoIL), a new PEFT method for VLP models.
• Formulate PEFT as a weight imitation learning process to directly optimize the low-rank adaptation approximation error bound.
• Introduce a hybrid approximation function to simplify the learning process for low-rank adaptations.

Main Results:

• MoIL significantly improves the optimization efficiency of low-rank parameters in VLP models.
• MoIL demonstrates superior performance and efficiency compared to existing PEFT methods across various VLP models and tasks.
• Achieved +2.3% improvement on image-text matching task by updating only 6.23% of parameters, outperforming full fine-tuning.
• Validated MoIL's inference efficiency and generalization capabilities on models like VLMO and VinVL.

Conclusions:

• MoIL offers a highly effective and efficient solution for adapting large VLP models.
• The proposed method significantly reduces the parameter and storage requirements for VLP model fine-tuning.
• MoIL sets a new benchmark for PEFT in vision-language tasks, enhancing both performance and efficiency.