Noise-Robust Vision-Language Pre-Training With Positive-Negative Learning.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Vision-Language Pre-training (VLP) learns joint image-text representations.
• Existing VLP methods are susceptible to the Noisy Correspondence (NC) problem from misaligned data.
• NC significantly degrades performance in downstream tasks, necessitating pre-training solutions.

Purpose of the Study:

• To investigate the impact of Noisy Correspondence (NC) on Vision-Language Pre-training (VLP) models.
• To develop a robust VLP method that mitigates the performance degradation caused by NC.
• To propose an objective-customized solution for enhancing NC robustness in VLP.

Main Methods:

• Proposed a novel NoisE-robust Vision-languagE pRe-training (NEVER) method.
• Developed a progressive and adaptive data splitting strategy for clean and noisy subsets.
• Employed positive learning (PL) and negative learning (NL) with a twin momentum model for target smoothing and sharpening.

Main Results:

• Empirically studied the influence of NC on VLP models.
• Demonstrated that NC significantly degrades downstream task performance.
• Showcased that the influence of NC varies across different pre-training objectives.

Conclusions:

• The proposed NEVER method effectively enhances VLP model robustness against NC.
• NEVER improves performance on various vision-language tasks.
• Handling NC during pre-training is crucial for robust VLP models.