Bridging multimedia heterogeneity gap via Graph Representation Learning for cross-modal retrieval

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Cross-modal retrieval faces challenges due to the "heterogeneity gap" caused by inconsistent feature representations across different data modalities.
• Existing methods often project data into a common space, requiring high model fitting ability.

Purpose of the Study:

• To propose a novel Graph Representation Learning (GRL) method to bridge the "heterogeneity gap" in cross-modal retrieval.
• To offer an alternative approach that avoids direct feature projection into an aligned space.

Main Methods:

• Developed a GRL approach comprising Feature Transfer Learning Network (FTLN) and Graph Representation Learning Network (GRLN).
• FTLN maps each modality to a latent space using cosine similarity.
• A cross-modal graph reconstructs data and relationships, with graph vertices embedded into a common space.

Main Results:

• The GRL method effectively bridges the "heterogeneity gap" by using a cross-modal graph as an intermediary.
• Extensive experiments on six datasets demonstrate superior performance compared to state-of-the-art cross-modal retrieval methods.

Conclusions:

• The proposed GRL method provides a simple yet effective solution for cross-modal retrieval challenges.
• Graph representation learning offers a promising direction for addressing modality heterogeneity in information retrieval.