Cross-Modal Retrieval With Partially Mismatched Pairs.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Cross-modal retrieval systems often suffer performance degradation due to partially mismatched pairs (PMPs) in real-world datasets.
• Existing methods struggle to effectively handle the noise introduced by PMPs, leading to suboptimal retrieval accuracy.

Purpose of the Study:

• To develop a robust cross-modal learning framework (RCL) that mitigates the negative impact of PMPs on retrieval performance.
• To introduce an unbiased estimator for cross-modal retrieval risk, enhancing model resilience against noisy data.

Main Methods:

• Proposes a unified theoretical framework, Robust Cross-modal Learning (RCL), incorporating an unbiased estimator of cross-modal retrieval risk.
• Employs a novel complementary contrastive learning paradigm to address overfitting and underfitting issues caused by PMPs.
• Leverages all available negative pairs to strengthen supervision and minimizes risk upper bounds to focus on hard samples.

Main Results:

• Demonstrates significant improvements in cross-modal retrieval performance across five benchmark datasets.
• Validates the effectiveness and robustness of the proposed RCL framework against state-of-the-art approaches.
• Achieves superior results in both image-text and video-text retrieval tasks.

Conclusions:

• The proposed Robust Cross-modal Learning (RCL) framework effectively enhances cross-modal retrieval by robustly handling partially mismatched pairs.
• RCL offers a promising direction for developing more reliable and accurate cross-modal retrieval systems in real-world applications.
• The complementary contrastive learning strategy proves effective in balancing model fitting and generalization for noisy datasets.