Bidirectional Relationship Inferring Network for Referring Image Localization and Segmentation.

Area of Science:

• Computer Vision
• Natural Language Processing
• Artificial Intelligence

Background:

• Referring image localization and segmentation are critical tasks in computer vision.
• Existing methods struggle to capture the intricate interdependence between linguistic descriptions and visual content.
• A need exists for models that can effectively integrate and align information from both modalities.

Purpose of the Study:

• To develop a novel network, the Bidirectional Relationship Inferring Network (BRINet), to address the limitations in current referring image understanding.
• To explicitly model the interdependence between language and vision for improved localization and segmentation accuracy.
• To enhance the alignment of cross-modal features through mutual guidance.

Main Methods:

• Proposed a Bidirectional Cross-Modal Attention Module (BCAM) comprising vision-guided linguistic attention and language-guided visual attention.
• Introduced an asymmetric language-guided visual attention to reduce computational costs.
• Utilized a segmentation-guided bottom-up augmentation module (SBAM) for multilevel information integration in object localization.

Main Results:

• The proposed BRINet significantly outperforms state-of-the-art methods.
• Achieved superior performance on three referring image localization datasets.
• Demonstrated effectiveness on four referring image segmentation datasets.

Conclusions:

• BRINet effectively addresses the challenge of language-vision interdependence in referring image tasks.
• The BCAM module facilitates better cross-modal feature alignment.
• The method shows strong generalization capabilities across various datasets for both localization and segmentation.