Unified semantic space learning for cross-modal retrieval.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Cross-modal retrieval is crucial for handling the growing volume of multimodal internet data.
• Graph convolutional networks (GCNs) have advanced cross-modal retrieval by incorporating sample and label correlations.
• Existing GCN methods often overlook correlations among semantic contents and underutilize instance-semantic content similarity.

Purpose of the Study:

• To propose a Unified Semantic Space Learning (USSL) method to address limitations in current cross-modal retrieval techniques.
• To explore correlations among semantic contents and map diverse data modalities (images, texts, labels) into a unified semantic space.
• To enhance similarity calculations between samples and between samples and semantic contents.

Main Methods:

• Constructing a label-multi-label graph to learn semantic content correlations using a Group Semantic Sharing Graph Convolutional Network.
• Developing an isomorphic InfoNCE loss to bridge the heterogeneity gap between samples and semantic contents.
• Implementing intra-modality and inter-modality InfoNCE losses to preserve semantic and structural consistencies in modality-invariant representations.

Main Results:

• Demonstrated the superiority of the proposed USSL method through comparative experiments.
• Successfully mapped images, texts, labels, and multi-labels into a unified semantic space.
• Effectively utilized semantic content correlations and instance-semantic content similarity for improved retrieval.

Conclusions:

• The proposed USSL method significantly advances cross-modal retrieval by effectively leveraging semantic content correlations.
• The unified semantic space facilitates more accurate similarity computations across different data modalities.
• USSL offers a promising approach for future research in multimodal data analysis and retrieval.