Aleatoric-Epistemic Joint Uncertainty Modeling for Cross-Modal Retrieval

Area of Science:

• Artificial Intelligence
• Computer Vision
• Natural Language Processing

Background:

• Cross-modal retrieval leverages large-scale vision-language pretraining models like CLIP.
• Existing methods map modalities to a shared space but struggle with data and model uncertainties, leading to unreliable results.
• Uncertainties stem from data ambiguity and noisy data-model pairs.

Purpose of the Study:

• To propose a novel cross-modal retrieval framework, AEUM, for reliable uncertainty modeling.
• To address both data (aleatoric uncertainty) and model (epistemic uncertainty) uncertainties.
• To improve the accuracy of cross-modal retrieval by correcting initial similarity scores.

Main Methods:

• AEUM jointly models aleatoric uncertainty (AU) and epistemic uncertainty (EU).
• AU is estimated using learnable semantic tokens for each modality to gauge data-induced uncertainty.
• EU is estimated via evidential learning to enhance robustness against noisy data.

Main Results:

• The proposed AEUM framework demonstrates effectiveness and generalization across multiple benchmarks.
• Experiments were conducted on seven datasets: five video-text (MSRVTT, LSMDC, MSVD, VATEX, DiDeMo) and two image-text (MSCOCO, Flickr30K).
• The method significantly improves retrieval accuracy by accounting for uncertainties.

Conclusions:

• AEUM provides reliable uncertainty estimation for improved cross-modal retrieval.
• The joint modeling of aleatoric and epistemic uncertainties leads to more accurate and robust retrieval results.
• The framework shows strong performance on diverse video-text and image-text retrieval tasks.