Toward Reasoning-Centric Video Object Segmentation via Multi-Modal Large Language Models

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Referring Video Object Segmentation (RVOS) typically relies on explicit object descriptions.
• Existing methods struggle with complex instructions requiring temporal understanding and relational reasoning.

Purpose of the Study:

• To develop a reasoning-centric RVOS model, RViSeg, capable of handling complex user queries.
• To enable Multi-modal Large Language Models (MLLMs) to perform efficient pixel-level video perception for RVOS.

Main Methods:

• Introduced a Spatial Token Merge (STM) module to consolidate video tokens into compact, region-level clusters while retaining spatial details.
• Developed a Query-based Target Retrieval (QTR) module using learnable tokens for mask prediction and spatio-temporal consistency.
• Constructed the InstructVideo benchmark for training and evaluating reasoning-centric RVOS.

Main Results:

• RViSeg effectively leverages MLLMs for complex RVOS tasks by processing structured visual information.
• The STM and QTR modules significantly improve segmentation accuracy and spatio-temporal consistency.
• Experiments validate the effectiveness of the proposed components and the InstructVideo benchmark.

Conclusions:

• RViSeg represents a significant advancement in RVOS by integrating MLLM reasoning capabilities.
• The proposed modules and benchmark facilitate more robust and accurate video object segmentation based on complex instructions.