Continual Instruction Tuning for Large Multimodal Models.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Vision

Background:

• Instruction tuning aligns large multimodal models (LMMs) with human intent via multi-task joint training.
• Exhaustive training on all emerging vision-language tasks is impractical, necessitating more efficient methods like continual learning.

Purpose of the Study:

• To investigate catastrophic forgetting in LMMs during continual instruction tuning.
• To evaluate the applicability of existing continual learning methods to LMM instruction tuning.
• To develop novel strategies for effective continual learning in LMMs.

Main Methods:

• Established the first benchmark for continual instruction tuning of LMMs.
• Integrated and adapted traditional continual learning approaches.
• Explored task-similarity dynamics to inform regularization and model expansion.

Main Results:

• Confirmed catastrophic forgetting in LMMs under continual instruction tuning.
• Demonstrated varying effectiveness of traditional continual learning methods.
• Showcased improved model performance using task-similarity-informed strategies.

Conclusions:

• Continual learning is crucial for adapting LMMs to evolving vision-language tasks.
• Existing continual learning methods can be adapted, but task-specific strategies are beneficial.
• Novel methods leveraging task similarity effectively mitigate forgetting and enhance LMM capabilities.