Improvement of deep cross-modal retrieval by generating real-valued representation

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Cross-modal retrieval (CMR) facilitates flexible data searching across different modalities.
• A key challenge in CMR is the heterogeneity gap, stemming from diverse statistical properties of multi-modal data.
• Representation learning, creating a common subspace, is a primary approach to address this gap.

Purpose of the Study:

• To propose a novel framework, Improvement of Deep Cross-Modal Retrieval (IDCMR), for enhancing cross-modal retrieval.
• To generate real-valued representations that effectively bridge the heterogeneity gap.
• To preserve both intra-modal and inter-modal similarity within the learned representations.

Main Methods:

• The IDCMR framework employs tailored training models for text and image modalities to maintain intra-modal similarity.
• Inter-modal similarity is preserved by minimizing a modality-invariance loss function.
• Performance is evaluated using the mean average precision (mAP) metric.

Main Results:

• IDCMR demonstrates superior performance compared to state-of-the-art methods in cross-modal retrieval tasks.
• The framework achieved relative improvements of 4% and 2% in mAP for text-to-image and image-to-text retrieval, respectively.
• Experiments were conducted on the MSCOCO and Xmedia datasets.

Conclusions:

• The proposed IDCMR framework effectively addresses the heterogeneity gap in cross-modal retrieval.
• IDCMR significantly improves retrieval accuracy by preserving essential similarities across modalities.
• The method offers a promising advancement for multi-modal data retrieval systems.