Rethinking and Improving Feature Pyramids for One-Stage Referring Expression Comprehension.

Area of Science:

• Computer Vision
• Natural Language Processing
• Artificial Intelligence

Background:

• Referring Expression Comprehension (REC) is crucial for cross-modal tasks like VQA and image retrieval.
• One-stage REC methods, treating it as language-conditioned object detection, often neglect multi-level feature integration, relying on single-scale features.
• Existing mature neck structures from object detection show limited impact on REC due to semantic ambiguity in coarse-grained fusion.

Purpose of the Study:

• To investigate and enhance feature pyramid utilization for one-stage Referring Expression Comprehension (REC).
• To address the limitations of current feature fusion strategies in REC, particularly the semantic ambiguity issue.
• To propose a novel method, Language-Guided FPN (LG-FPN), for more effective multi-level feature integration in REC.

Main Methods:

• Developed a novel Language-Guided FPN (LG-FPN) incorporating language-gate and union-gate mechanisms.
• Dynamically allocated and selected fine-grained information through stacked gates for improved feature fusion.
• Conducted extensive contrastive and ablative experiments to validate the effectiveness of LG-FPN.

Main Results:

• Demonstrated that mature neck structures like FPN have limited impact on REC performance.
• Identified semantic ambiguity as a key issue in coarse-grained feature fusion strategies.
• LG-FPN proved to be an effective and adaptable module across different visual backbones, fusion strategies, and detection heads.

Conclusions:

• The proposed LG-FPN significantly improves performance in one-stage Referring Expression Comprehension.
• LG-FPN offers a robust solution to the semantic ambiguity problem in feature fusion for REC.
• The method achieves state-of-the-art results on four benchmark referring expression datasets.