Cross-Modal Progressive Comprehension for Referring Segmentation.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Natural Language Processing

Background:

• Referring segmentation aims to identify specific objects in images/videos based on natural language descriptions.
• Existing methods often use a one-stage approach for feature fusion, which may not fully capture human-like progressive reasoning.

Purpose of the Study:

• To propose a novel cross-modal progressive comprehension (CMPC) scheme to enhance referring segmentation models.
• To improve both referring image segmentation and referring video segmentation by mimicking human cognitive processes.

Main Methods:

• Developed CMPC-I (Image) module using entity, attribute, and relational words with spatial graph reasoning.
• Developed CMPC-V (Video) module extending CMPC-I with action words and temporal graph reasoning.
• Introduced a Text-Guided Feature Exchange (TGFE) module for multi-level feature refinement.

Main Results:

• The proposed CMPC scheme, combined with TGFE, achieved new state-of-the-art performance on multiple referring image and video segmentation benchmarks.
• The progressive approach effectively locates and distinguishes target entities based on linguistic cues.

Conclusions:

• The CMPC scheme offers a more effective way to perform referring segmentation by simulating human-like progressive comprehension.
• The integration of CMPC and TGFE modules significantly advances the capabilities of referring segmentation models for both images and videos.