Pseudo-Text Guided Robust Learning for Noisy Correspondence in Cross-Modal Retrieval

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Noisy Correspondence (NC) in multimedia datasets challenges cross-modal retrieval.
• Existing methods degrade significantly with increasing noise levels.

Purpose of the Study:

• Introduce Pseudo-Text guided Robust Learning (PTRL) to enhance model robustness against NC.
• Improve cross-modal retrieval performance under high noise conditions.

Main Methods:

• Leverage pseudo-text as supervisory signals for identifying noisy pairs.
• Employ a data division criterion to distinguish clean from noisy data.
• Utilize a pseudo-text replacement strategy for semantic consistency and data augmentation.
• Incorporate a robust InfoNCE loss to stabilize training and mitigate overfitting.

Main Results:

• PTRL achieves state-of-the-art performance and robustness.
• Demonstrated significant improvements on Flickr30K (+60.1% rSum) and MS-COCO (+22.6% rSum) at 80% noise.
• Outperformed existing methods substantially in high-noise scenarios.

Conclusions:

• PTRL effectively addresses the challenge of Noisy Correspondence in cross-modal retrieval.
• The proposed framework offers superior robustness and generalization capabilities.
• Pseudo-text guidance and robust loss functions are key to overcoming noise in multimedia datasets.