Temporal Bayesian Fusion for Affect Sensing: Combining Video, Audio, and Lexical Modalities

Area of Science:

• Affective computing
• Multimodal machine learning
• Human-computer interaction

Background:

• Affective states fluctuate during conversations and are expressed through various channels like facial expressions, voice, and speech.
• While unimodal affect sensing has advanced, emotion recognition remains challenging.
• Multimodal approaches combining different data sources are explored to improve affect recognition, but often yield limited benefits.

Purpose of the Study:

• To develop a temporal Bayesian fusion method for continuous, real-valued estimation of affect dimensions (valence, arousal, power, expectancy).
• To enhance emotion recognition accuracy by effectively integrating information from video, audio, and lexical modalities.

Main Methods:

• Developed a temporal Bayesian fusion framework for multimodal affect sensing.
• Integrated video, audio, and lexical data for affect dimension estimation.
• Employed a temporal prediction model as a prior in Bayesian fusion, incorporating uncertainties from unimodal predictions.

Main Results:

• Achieved substantial gains in affect recognition performance compared to prior methods.
• Demonstrated the effectiveness of using temporal correlations and biases for affect estimation, particularly at the start of conversations.
• The proposed model is simpler, avoiding inter-modal relationship modeling and using fewer interpretable parameters.

Conclusions:

• Temporal Bayesian fusion offers a significant improvement for multimodal affect sensing.
• The method's ability to handle temporal dynamics and uncertainties enhances continuous affect dimension estimation.
• This approach provides a more effective and interpretable solution for emotion recognition in conversations.