Referring Segmentation via Encoder-Fused Cross-Modal Attention Network.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Referring segmentation aims to identify image regions based on natural language descriptions.
• Current methods often process multi-modal interactions late in the network, neglecting multi-scale feature correlations.
• This limits the ability to effectively link linguistic concepts to visual elements across different granularities.

Purpose of the Study:

• To propose a novel Encoder Fusion Network (EFN) for referring segmentation.
• To shift multi-modal feature learning to the encoding stage for gradual language-guided refinement.
• To improve the accuracy and detail-awareness of segmentation in both images and videos.

Main Methods:

• Developed an Encoder Fusion Network (EFN) that integrates visual and linguistic features early in the encoding process.
• Employed a co-attention mechanism for enhanced alignment between visual and language modalities.
• Introduced a Boundary Enhancement Module (BEM) for detailed segmentation and an Asymmetric Cross-Frame Attention Module (ACFM) for video temporal information.

Main Results:

• The proposed EFN method achieved state-of-the-art performance on benchmark referring image and video segmentation datasets.
• Experiments demonstrated the effectiveness of the co-attention mechanism and the boundary enhancement module.
• The ACFM module proved successful in capturing crucial temporal dynamics for video segmentation.

Conclusions:

• The Encoder Fusion Network (EFN) offers a more effective approach to referring segmentation by leveraging early multi-modal fusion.
• The integration of co-attention, BEM, and ACFM significantly boosts segmentation accuracy and temporal understanding.
• This work advances the field by providing a robust framework for language-guided visual segmentation.