Quantifying Epistemic Uncertainty in Multimodal Long-Tailed Classification: A Belief Entropy-Based Evidential Fusion...

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Vision

Background:

• Deep multimodal learning excels across vision, language, and audio.
• Real-world data often exhibits long-tailed distributions, degrading performance on tail classes.
• Existing fusion schemes inadequately address modality-specific uncertainty and class-level fairness.

Purpose of the Study:

• To present a novel framework, Uncertainty-Quantified Multimodal Learning for Long-Tailed Classification (UMuLT), integrating evidential reasoning with deep learning.
• To tackle information discrepancies and performance degradation in long-tailed multimodal classification.
• To enhance fairness and accuracy for underrepresented classes.

Main Methods:

• Developed an uncertainty-gated evidential fusion module to down-weight unreliable modalities.
• Incorporated an exponential moving average (EMA) fairness regularizer to amplify tail-class gradients.
• Implemented a two-stage cross-modal consistency regularizer: tail specialization and end-to-end fine-tuning.

Main Results:

• UMuLT demonstrated consistent gains over strong baselines on multimodal classification benchmarks.
• Significant improvements were observed in overall metrics, model calibration, and performance on tail subsets.
• Statistical significance tests confirmed the superiority of the proposed UMuLT framework.

Conclusions:

• The UMuLT framework effectively addresses modality uncertainty and class imbalance in long-tailed multimodal learning.
• The proposed methods provide a practical solution for improving fairness and performance on tail classes.
• UMuLT offers a robust approach for real-world multimodal classification tasks with imbalanced data.