Cross-Modal Retrieval With Noisy Correspondence via Consistency Refining and Mining.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Existing cross-modal retrieval (CMR) methods often fail due to noisy correspondence (NC) in data annotations.
• NC arises from errors during data collection or annotation, compromising the reliability of paired data.

Purpose of the Study:

• To propose a novel method, CREAM (Consistency REfining And Mining), to address the challenge of noisy correspondence in CMR.
• To leverage the distinction between correspondence and consistency to improve CMR model robustness.

Main Methods:

• CREAM utilizes a collaborative learning paradigm to detect and correct positive correspondences.
• A negative mining approach is employed to exploit consistency within potentially mislabeled data.
• The method differentiates between true/false positives and true/false negatives based on correspondence and consistency alignment.

Main Results:

• CREAM effectively prevents overfitting on false positives and utilizes consistency from false negatives.
• Experiments on Flickr30K, MS-COCO, and Conceptual Captions benchmarks demonstrate significant improvements in CMR.
• The method's robustness was further validated in graph matching tasks against fine-grained NC.

Conclusions:

• CREAM offers a robust solution to the noisy correspondence problem in cross-modal retrieval.
• The proposed consistency refining and mining strategy enhances model performance and reliability.
• The approach shows broad applicability, extending beyond image-text retrieval to other domains like graph matching.