FM-LoRA: Factorized Low-Rank Meta-Prompting for Continual Learning.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Vision

Background:

• Continual learning (CL) aims to adapt pre-trained models for sequential tasks without catastrophic forgetting.
• Existing CL methods often suffer from parameter expansion and lack task similarity awareness, hindering performance.
• Transformers are powerful pre-trained models but require efficient adaptation strategies for sequential learning.

Purpose of the Study:

• To propose FM-LoRA, a novel and efficient low-rank adaptation method for continual learning.
• To address parameter growth and improve task similarity awareness in continual adaptation.
• To enable learning generalizable models across diverse sequential tasks and domains.

Main Methods:

• FM-LoRA integrates a dynamic rank selector (DRS) and dynamic meta-prompting (DMP).
• It leverages a shared low-rank subspace to preserve knowledge and allocate model capacity effectively.
• The method was evaluated on class-incremental learning (CIL) and domain-incremental learning (DIL) benchmarks using Transformers.

Main Results:

• FM-LoRA demonstrated effective mitigation of catastrophic forgetting in continual learning.
• The method achieved robust performance across diverse tasks and domains, including ImageNet-R, CIFAR100, CUB200, and DomainNet.
• FM-LoRA avoids continual parameter expansion, offering an efficient solution for sequential adaptation.

Conclusions:

• FM-LoRA provides an efficient and effective approach to continual learning for sequential tasks.
• The proposed method enhances model generalizability across different classes and domains.
• FM-LoRA offers a sustainable solution for adapting large pre-trained models in dynamic learning environments.