Revisiting Image-Language Networks for Open-Ended Phrase Detection

Area of Science:

• Computer Vision
• Artificial Intelligence
• Natural Language Processing

Background:

• Existing phrase grounding methods assume phrase relevance to images.
• Realistic scenarios require identifying and localizing phrases within images.
• This task generalizes object detection to open-ended vocabularies, incorporating few- and zero-shot learning.

Purpose of the Study:

• To develop a robust method for phrase grounding that addresses both relevance identification and localization.
• To enhance the capabilities of object detection for open-ended vocabularies.
• To improve few- and zero-shot detection performance.

Main Methods:

• The study extends the Faster R-CNN architecture to effectively relate image regions with natural language phrases.
• Canonical Correlation Analysis (CCA) is employed for careful initialization of the network's classification layers.
• This initialization encourages more discerning reasoning between semantically similar phrases.

Main Results:

• The proposed approach achieves over double the performance compared to naive adaptations.
• The method demonstrates strong performance across three diverse phrase grounding datasets: Flickr30K Entities, ReferIt Game, and Visual Genome.
• Effective handling of large test-time phrase vocabulary sizes (5K, 32K, and 159K) was achieved.

Conclusions:

• The developed method offers a significant advancement in realistic natural language phrase grounding.
• The approach provides a more discerning and accurate solution for open-vocabulary, few- and zero-shot detection.
• Initialization using CCA is crucial for improving phrase reasoning and overall performance.