CRISP: Contrastive Residual Injection and Semantic Prompting for Continual Video Instance Segmentation

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Continual video instance segmentation (CVIS) demands models that can learn new categories without forgetting previous knowledge.
• Maintaining temporal consistency of instances across video frames is a critical challenge in CVIS.
• Existing methods struggle with instance-wise, category-wise, and task-wise confusion during incremental learning.

Purpose of the Study:

• To introduce a new framework, Contrastive Residual Injection and Semantic Prompting (CRISP), to tackle the complexities of CVIS.
• To enhance both plasticity (learning new categories) and stability (retaining old knowledge) in CVIS models.
• To improve temporal consistency and reduce catastrophic forgetting in long-term CVIS tasks.

Main Methods:

• CRISP employs instance tracking and an instance correlation loss for instance-wise learning, focusing on query space correlation and task specificity.
• An adaptive residual semantic prompt (ARSP) learning framework with a query-prompt matching mechanism is used for category-wise learning.
• Contrastive learning and a semantic consistency loss maintain semantic coherence, while a prompt initialization strategy addresses task-wise learning.

Main Results:

• CRISP significantly outperforms existing continual segmentation methods on YouTube-VIS-2019 and YouTube-VIS-2021 datasets.
• The framework effectively avoids catastrophic forgetting, a common issue in continual learning.
• Demonstrated improvements in both segmentation and classification performance in long-term CVIS.

Conclusions:

• CRISP provides an effective solution for the challenges in continual video instance segmentation.
• The proposed methods successfully balance plasticity and stability, crucial for incremental learning.
• CRISP offers a promising direction for advancing long-term continual video instance segmentation research.