Modality-specific Cross-modal Similarity Measurement with Recurrent Attention Network

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Cross-modal retrieval is crucial for accessing information across diverse data types like images and text.
• Existing methods often lose modality-specific details by projecting data into a common space.
• The imbalanced and complementary nature of information between modalities presents a key challenge.

Purpose of the Study:

• To propose a modality-specific cross-modal similarity measurement (MCSM) approach.
• To overcome the limitations of common representation methods in cross-modal retrieval.
• To effectively leverage both modality-specific characteristics and inter-modal relationships.

Main Methods:

• Developed an end-to-end framework generating modality-specific cross-modal similarity without a common representation.
• Utilized recurrent attention networks to exploit modality-specific features within independent semantic spaces.
• Employed attention-based joint embedding for projecting data and guiding fine-grained cross-modal correlation.
• Implemented adaptive fusion of modality-specific similarities to enhance retrieval performance.

Main Results:

• The proposed MCSM approach demonstrated superior performance in cross-modal retrieval tasks.
• Experiments on Wikipedia, Pascal Sentence, MS-COCO, and XMediaNet datasets validated the method's effectiveness.
• The approach successfully captured imbalanced and complementary relationships between different data modalities.

Conclusions:

• The modality-specific approach effectively preserves unique data characteristics while learning cross-modal correlations.
• MCSM offers a significant advancement in cross-modal retrieval by addressing information imbalance and complementarity.
• The method provides a robust and effective solution for retrieving information across diverse data modalities.