Cross-modal Guided Visual Representation Learning for Social Image Retrieval.

Area of Science:

• Computer Science
• Information Retrieval
• Machine Learning

Background:

• Social images possess rich but noisy tags from user contributions, hindering effective image retrieval.
• Existing cross-modal learning methods struggle with isolated, noisy tags and unimodal retrieval requirements.

Purpose of the Study:

• To develop a framework for precise unimodal social image retrieval using noisy tags.
• To effectively filter noise and exploit cross-modal correlations for accurate image representation learning.

Main Methods:

• Proposed Cross-modal Guided Visual Representation (CGVR) framework.
• Extracted accurate semantic image representations from noisy tags.
• Transferred representation learning ability to an image-only hashing subnetwork.
• Embedded common-sense tag relationships into attention for joint context awareness.

Main Results:

• CGVR achieved significant improvements in Mean Average Precision (MAP) over state-of-the-art methods.
• Demonstrated approximately 8.82 and 5.45 points MAP improvement on social image benchmarks.
• Established CGVR as a new baseline for social image retrieval.

Conclusions:

• CGVR effectively addresses the challenge of noisy tags in social image retrieval.
• The framework successfully learns precise visual representations by leveraging cross-modal interactions.
• CGVR offers a robust solution for unimodal retrieval from noisy social image data.