Context-Infused Trajectories: Enhancing Context and Frame Consistency in Reasoning Video Object Segmentation.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Reasoning video object segmentation (ReaVOS) faces challenges with limited frame compression and global segmentation tokens.
• Existing multimodal large language models (MLLMs) struggle with spatial detail loss and inconsistent segmentation.

Purpose of the Study:

• To develop a novel framework, Context-infused Consistent Video Segmentor (CiCVS), for accurate and temporally coherent video object segmentation.
• To enhance contextual guidance and preserve spatial details in ReaVOS.

Main Methods:

• Implemented Hierarchical Frame Sampling (HFS) for broad temporal coverage and target frame selection.
• Introduced Contextual Token Prompting (CTP) using support frames to guide MLLMs for specialized token generation.
• Developed the Multimodal Injection Compressor (MIC) block for efficient integration of frame features and text semantics.

Main Results:

• CiCVS achieved state-of-the-art performance on multiple benchmarks.
• Significant improvements in J&F scores were observed: +2.7 on CoCoRVOS, +1.4 on ReVOS, and +7.0 on ReasonVOS.
• Demonstrated superior contextual reasoning and segmentation capabilities.

Conclusions:

• CiCVS effectively addresses limitations in current ReaVOS methods by incorporating contextual information.
• The proposed framework enhances temporal consistency and segmentation accuracy in complex video sequences.
• Introduced the CoCoRVOS benchmark to further advance research in temporally intricate video reasoning.