Noisy Correspondence Rectification in Multimodal Clustering Space for Cross-Modal Matching

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Cross-modal matching is crucial for projecting diverse sensory data into a shared feature space.
• Training requires large, accurately aligned multimodal datasets, which are difficult and costly to obtain.
• Internet-sourced datasets often contain mismatched pairs, degrading model performance.

Purpose of the Study:

• To propose BiCro++ (Improved Bidirectional Cross-modal Similarity Consistency), a module enhancing existing cross-modal matching models.
• To improve model robustness against noisy data using self-adaptive soft labels.
• To leverage bidirectional similarity consistency as a self-supervision signal.

Main Methods:

• BiCro++ integrates into existing models, generating dynamic soft labels reflecting true data correspondences.
• Employs Diagonal-Dominance Purification to identify reliable data anchors from noisy sets.
• Utilizes Hybrid-level Codebook Alignment for enhanced bidirectional cross-modal similarity consistency.

Main Results:

• The method significantly improves the noise-robustness of various cross-modal matching models.
• BiCro++ surpasses state-of-the-art methods, achieving average recall improvements of 5.3%, 3.1%, and 6.4% on three datasets.
• Demonstrates effective handling of mismatched data pairs common in large-scale datasets.

Conclusions:

• BiCro++ offers an effective solution for training robust cross-modal matching models with noisy datasets.
• The proposed self-adaptive soft label strategy and purification mechanisms are key to its success.
• This approach advances the field by enabling better utilization of readily available, albeit imperfect, multimodal data.