Causal Prompts for Open-Vocabulary Video Instance Segmentation.

Area of Science:

• Computer Vision
• Artificial Intelligence

Background:

• Open-vocabulary video instance segmentation aims to detect, segment, and track objects, including unknown categories.
• Current methods often fail to utilize temporal information from previous frames, hindering generalization in open-world scenarios.

Purpose of the Study:

• To propose CPOVIS, a novel framework that enhances temporal reasoning and semantic consistency for open-vocabulary video instance segmentation.
• To leverage causal prompts dynamically propagated from historical frames to improve performance on unseen object categories.

Main Methods:

• CPOVIS utilizes a Mask2Former architecture with a CLIP backbone, incorporating PromptCLIP for cross-modal alignment.
• Key innovations include a Visual Prompt Injector for spatial-temporal coherence and a Taxonomy Prompt Infuser for semantic consistency.
• A contrastive learning strategy and adaptation of Segment Anything Model (SAM2) are employed to boost segmentation and tracking capabilities.

Main Results:

• CPOVIS achieves state-of-the-art performance on seven challenging open- and closed-vocabulary video segmentation benchmarks.
• The framework significantly outperforms existing methods in detecting, segmenting, and tracking objects, especially novel categories.
• Causal prompt propagation is demonstrated to be crucial for advancing video understanding in open-world settings.

Conclusions:

• CPOVIS effectively addresses the limitations of existing methods by incorporating causal temporal cues.
• The proposed framework demonstrates robust open-world generalization capabilities for video instance segmentation.
• This work highlights the importance of causal prompt propagation for improving video analysis and object recognition in dynamic, open-world environments.