IAP: Improving Continual Learning of Vision-Language Models via Instance-Aware Prompting | JoVE Visualize

Area of Science:

Artificial Intelligence
Computer Vision
Machine Learning

Background:

Pre-trained vision-language models (PT-VLMs) encounter challenges in multi-domain task incremental learning (MTIL), particularly forward and backward forgetting, due to the incremental arrival of tasks without access to prior data.
Parameter-efficient fine-tuning (PEFT) techniques, like prompt tuning, are used to adapt PT-VLMs but existing methods overlook the critical role of PEFT parameter settings, specifically prompt design.

Purpose of the Study:

To address the limitations of existing methods in optimizing prompt designs for diverse tasks within the MTIL scenario.
To propose a novel framework, Instance-Aware Prompting (IAP), that enhances adaptation to new tasks while mitigating catastrophic forgetting in PT-VLMs.

Main Methods:

Developed Instance-Aware Gated Prompting (IA-GP) to adaptively assign prompts across transformer layers at the instance level, improving new task adaptation and reducing forgetting.
Introduced Instance-Aware Class-Distribution-Driven Prompting (IA-CDDP) to enhance task adaptation by calculating instance-level confidence scores related to task labels.

Main Results:

Experimental evaluations on 11 datasets demonstrated the effectiveness of the proposed Instance-Aware Prompting (IAP) framework.
The IA-GP and IA-CDDP strategies significantly improved performance in the MTIL setting, showcasing superior adaptation and reduced forgetting compared to existing approaches.
The method achieved strong results across three standard performance metrics, validating its efficacy.

Conclusions:

The proposed Instance-Aware Prompting (IAP) framework effectively optimizes prompt designs for PT-VLMs in MTIL scenarios.
IA-GP and IA-CDDP offer a robust solution for enhancing new task adaptation and mitigating forgetting in memory-constrained incremental learning settings.
The findings highlight the importance of instance-level prompt optimization for advancing PT-VLM capabilities in dynamic learning environments.