IAP: Improving Continual Learning of Vision-Language Models via Instance-Aware Prompting.

Area of Science:

• Artificial Intelligence
• Computer Vision
• Machine Learning

Background:

• Pre-trained vision-language models (PT-VLMs) encounter challenges in multi-domain task incremental learning (MTIL), particularly forward and backward forgetting, due to the incremental arrival of tasks without access to prior data.
• Parameter-efficient fine-tuning (PEFT) techniques, like prompt tuning, are used to adapt PT-VLMs but existing methods overlook the critical role of PEFT parameter settings, specifically prompt design.

Purpose of the Study:

• To address the limitations of existing methods in optimizing prompt designs for diverse tasks within the MTIL scenario.
• To propose a novel framework, Instance-Aware Prompting (IAP), that enhances adaptation to new tasks while mitigating catastrophic forgetting in PT-VLMs.

Main Methods:

• Developed Instance-Aware Gated Prompting (IA-GP) to adaptively assign prompts across transformer layers at the instance level, improving new task adaptation and reducing forgetting.
• Introduced Instance-Aware Class-Distribution-Driven Prompting (IA-CDDP) to enhance task adaptation by calculating instance-level confidence scores related to task labels.

Main Results:

• Experimental evaluations on 11 datasets demonstrated the effectiveness of the proposed Instance-Aware Prompting (IAP) framework.
• The IA-GP and IA-CDDP strategies significantly improved performance in the MTIL setting, showcasing superior adaptation and reduced forgetting compared to existing approaches.
• The method achieved strong results across three standard performance metrics, validating its efficacy.

Conclusions:

• The proposed Instance-Aware Prompting (IAP) framework effectively optimizes prompt designs for PT-VLMs in MTIL scenarios.
• IA-GP and IA-CDDP offer a robust solution for enhancing new task adaptation and mitigating forgetting in memory-constrained incremental learning settings.
• The findings highlight the importance of instance-level prompt optimization for advancing PT-VLM capabilities in dynamic learning environments.