Decoupled Seg Tokens Make Stronger Reasoning Video Segmenter and Grounder.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Existing video segmentation and grounding methods fuse features directly, leading to entanglement of visual and semantic information.
• This entanglement degrades segmentation accuracy by mixing dynamic visual data with static semantics.

Purpose of the Study:

• To propose DeSa2VA, a novel decoupling-enhanced prompting scheme to mitigate feature entanglement in video segmentation.
• To address information processing limitations in SAM-2 through text pre-training and a linear decoupling module.

Main Methods:

• A pre-training paradigm converts textual labels into point-level prompts and generates corresponding text masks, refined by a hybrid loss function.
• A linear projection disentangles large language model hidden states into separate textual and visual feature subspaces.
• A dynamic mask fusion strategy combines decoupled features using triple supervision.

Main Results:

• DeSa2VA achieves state-of-the-art performance across various tasks, including image and video segmentation and question answering.
• The decoupling approach successfully mitigates the negative impact of feature entanglement on segmentation accuracy.
• Experimental results validate the effectiveness of the proposed text pre-training and linear decoupling modules.

Conclusions:

• DeSa2VA offers a systematic solution to information processing limitations in video segmentation models.
• The proposed method significantly improves segmentation accuracy and generalizes well to diverse tasks.
• This work advances the field of video understanding by enabling more precise and semantically grounded segmentation.