NOTO: Noise-Tolerate Evidential Learning for Open-Set Cross-Modal Retrieval

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Cross-modal retrieval (CMR) methods often rely on clean annotations and closed-set assumptions, which are unrealistic.
• Real-world data frequently contains label noise and emergent open-set categories, termed open-set noisy labels (OSNL).
• OSNL poses significant challenges for CMR by forcing models to learn incorrect semantic associations, degrading performance.

Purpose of the Study:

• To develop a robust framework for cross-modal retrieval (CMR) that can effectively handle both closed-set and open-set noisy labels (OSNL).
• To improve the learning of shared cross-modal representations in the presence of imperfect annotations.
• To enhance the performance and robustness of CMR systems in practical, real-world scenarios.

Main Methods:

• Proposes NOise-TOlerate evidential learning (NOTO), a novel framework for robust cross-modal representation learning.
• Introduces a Robust Evidential Learning (REL) module to detect and classify instances as clean, closed-set noisy, or open-set noisy using Dirichlet evidence and belief masses.
• Develops an Adaptive Noise-aware Contrast (ANC) module to select reliable positive pairs and maximize mutual information for improved cross-modal alignment.

Main Results:

• The NOTO framework demonstrates superior retrieval performance compared to ten state-of-the-art CMR methods.
• NOTO exhibits significant robustness against open-set noisy labels (OSNL).
• Experiments conducted on four benchmarks validate the effectiveness of the proposed approach.

Conclusions:

• The NOTO framework offers a robust solution for cross-modal retrieval (CMR) under challenging noisy label conditions, including open-set noisy labels (OSNL).
• The proposed REL and ANC modules effectively address the limitations of existing CMR methods in realistic data scenarios.
• NOTO significantly advances the state-of-the-art in robust cross-modal representation learning and retrieval.