Multi-Modal Mutual Attention and Iterative Interaction for Referring Image Segmentation.

Area of Science:

• Computer Vision
• Natural Language Processing
• Artificial Intelligence

Background:

• Referring image segmentation requires generating object masks from natural language descriptions.
• Existing Transformer-based methods struggle with effectively fusing multi-modal information, leading to vision-dominated features and mask prediction uncertainty.

Purpose of the Study:

• To enhance referring image segmentation by improving the fusion of language and vision modalities.
• To address the limitations of generic attention mechanisms in current Transformer models for multi-modal understanding.

Main Methods:

• Proposed Multi-Modal Mutual Attention (M3Att) and Multi-Modal Mutual Decoder (M3Dec) for improved cross-modal feature fusion.
• Introduced Iterative Multi-modal Interaction (IMI) for continuous language-vision feature exchange.
• Developed Language Feature Reconstruction (LFR) to preserve language information during feature extraction.

Main Results:

• The proposed M3Att and M3Dec methods significantly enhance baseline performance in referring image segmentation.
• The approach consistently outperforms state-of-the-art methods on the RefCOCO dataset series.

Conclusions:

• The novel M3Att, M3Dec, IMI, and LFR techniques offer a more robust solution for referring image segmentation.
• Effective multi-modal fusion is crucial for accurate object localization and mask generation from textual descriptions.