COLD Fusion: Calibrated and Ordinal Latent Distribution Fusion for Uncertainty-Aware Multimodal Emotion Recognition

Area of Science:

• Artificial Intelligence
• Machine Learning
• Affective Computing

Background:

• Emotion recognition from multimodal data (face, voice) is challenging due to inherent uncertainties in input data and labels.
• Existing machine learning frameworks often struggle to quantify and leverage these uncertainties effectively.

Purpose of the Study:

• To develop an uncertainty-aware multimodal fusion approach for improved emotion prediction.
• To quantify modality-wise aleatoric uncertainty for better emotion recognition performance.
• To enhance the robustness of emotion recognition models against novel noise patterns.

Main Methods:

• Proposed a novel fusion framework learning latent distributions over unimodal temporal contexts by constraining variance.
• Introduced variance constraints (Calibration and Ordinal Ranking) to represent modality informativeness for emotion recognition.
• Utilized a softmax distributional matching loss to jointly impose calibration and ordinal ranking constraints.

Main Results:

• The proposed method improved generalization performance in emotion recognition models across AVEC 2019 CES, CMU-MOSEI, and IEMOCAP datasets.
• Enhanced predictive uncertainty estimation, providing better insights into model confidence.
• Demonstrated increased robustness of the models to novel noise patterns encountered during testing.

Conclusions:

• The uncertainty-aware multimodal fusion approach effectively addresses challenges in emotion recognition.
• Quantifying and utilizing modality-wise uncertainty leads to more accurate and robust emotion prediction models.
• The method shows significant potential for real-world applications requiring reliable emotion analysis.