Toward Robust Referring Image Segmentation.

Area of Science:

• Computer Vision
• Natural Language Processing
• Artificial Intelligence

Background:

• Referring Image Segmentation (RIS) is a key vision-language task for generating object masks from text.
• Existing RIS models struggle with inaccurate or misleading text descriptions, termed negative sentences.
• A robust approach is needed to handle both accurate and inaccurate textual guidance.

Purpose of the Study:

• To introduce Robust Referring Image Segmentation (R-RIS) capable of processing negative sentences.
• To develop new datasets and evaluation metrics for R-RIS.
• To propose a novel transformer-based model, RefSegformer, for enhanced RIS.

Main Methods:

• Created three R-RIS datasets by augmenting existing RIS datasets with negative sentences.
• Developed unified metrics to evaluate performance on both positive and negative text inputs.
• Proposed RefSegformer, a transformer model with a token-based fusion module, adaptable for R-RIS.

Main Results:

• RefSegformer achieved state-of-the-art performance on established RIS benchmarks.
• The model demonstrated strong capabilities in the new R-RIS setting, handling negative sentences effectively.
• Established a new baseline for robust referring image segmentation.

Conclusions:

• The proposed R-RIS formulation and RefSegformer model offer a significant advancement in vision-language tasks.
• The approach effectively addresses the challenge of inaccurate textual descriptions in image segmentation.
• This work provides a robust foundation for future research in referring image segmentation.