Causality-Invariant Interactive Mining for Cross-Modal Similarity Learning.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Vision

Background:

• Learning consistent similarity across different data modalities is crucial but challenging.
• Existing methods struggle with the 'modality gap', leading to performance degradation in cross-modal tasks.

Purpose of the Study:

• To propose a novel cross-modal similarity learning method, Causality-Invariant Interactive Mining (CIIM).
• To effectively capture relationships among samples and modalities for consistent feature embeddings.
• To address the modality gap through sample-wise and feature-wise approaches.

Main Methods:

• CIIM utilizes sample-wise learning with single-modality and hybrid-modality proxies and metric losses.
• It incorporates causal intervention for feature-wise bias elimination and invariant embedding reconstruction.
• The method derives causality-invariant feature embeddings in a unified metric space.

Main Results:

• CIIM effectively captures informative relationships across samples and modalities.
• The method successfully bridges the modality gap.
• Experimental results demonstrate the superiority of CIIM over state-of-the-art methods on cross-modality tasks.

Conclusions:

• CIIM offers a robust solution for cross-modal similarity learning.
• The proposed method achieves superior performance by addressing the modality gap.
• CIIM generates modality-consistent and causality-invariant feature embeddings.