Learning Two-Branch Neural Networks for Image-Text Matching Tasks.

Area of Science:

• Computer Vision
• Machine Learning
• Artificial Intelligence

Background:

• Image-language matching is crucial for understanding visual content.
• Tasks include image-sentence retrieval and visual grounding (region-phrase matching).
• Existing methods require effective learning of similarities between visual and textual data.

Purpose of the Study:

• To investigate two-branch neural networks for learning image-language similarities.
• To propose novel network architectures for improved representation learning.
• To enhance performance in both retrieval and grounding tasks.

Main Methods:

• Developed an embedding network with a shared latent space using maximum-margin ranking loss and neighborhood constraints.
• Introduced a similarity network that fuses branches via element-wise product and uses regression loss.
• Implemented improved neighborhood sampling for mini-batch construction.

Main Results:

• Achieved high accuracy in phrase localization on the Flickr30K Entities dataset.
• Demonstrated strong performance in bi-directional image-sentence retrieval on Flickr30K and MSCOCO datasets.
• Both proposed network structures showed significant improvements.

Conclusions:

• The proposed two-branch neural networks are effective for image-language matching.
• Novel sampling strategies and network architectures enhance performance.
• The methods generalize well across different datasets and tasks.