GCMR-IMA: graph-based cross-modal retrieval with incomplete modality awareness.

Area of Science:

• Computer Science
• Artificial Intelligence
• Information Retrieval

Background:

• Cross-modal retrieval, particularly for image-text pairs, is crucial for diverse applications but faces challenges.
• Modality heterogeneity, missing data, and noise hinder traditional and deep learning methods.
• Existing deep learning approaches struggle with long-range semantic relationships and incomplete data.

Purpose of the Study:

• To develop a robust graph-based method for incomplete modality awareness cross-modal retrieval (GCMR-IMA).
• To address limitations in handling missing modalities and capturing long-range semantic dependencies.
• To enhance the effectiveness and robustness of cross-modal retrieval systems.

Main Methods:

• Proposed a dual-layer semantic architecture with image-text dual embedding for global semantic graph learning.
• Introduced a sparsified modality adjacency graph to model inter-modality relationships and enhance awareness.
• Implemented a graph-guided missing modality awareness method and an adaptive loss function for context-aware detection.

Main Results:

• GCMR-IMA demonstrates strong performance despite the presence of missing modalities.
• The method effectively captures long-range semantic relationships and handles modality heterogeneity.
• Experimental results validate the robustness and effectiveness of the proposed framework.

Conclusions:

• The GCMR-IMA framework significantly improves cross-modal retrieval, especially in scenarios with incomplete data.
• This approach offers enhanced robustness and effectiveness for real-world applications.
• The study contributes a novel solution for advanced image-text retrieval systems.