Gradient and Structure Consistency in Multimodal Emotion Recognition.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Affective Computing

Background:

• Multimodal emotion recognition integrates text, visual, and audio data.
• Current methods often overlook inter-modal differences, causing optimization bias and noise.
• This heterogeneity hinders effective joint learning in emotion AI.

Purpose of the Study:

• To propose a novel approach, Gradient and Structure Consistency (GSCon), for robust multimodal emotion recognition.
• To address optimization biases and interaction noise stemming from multimodal heterogeneity.
• To enhance the accuracy and reliability of inferring emotional states from diverse data sources.

Main Methods:

• GSCon employs a two-level strategy: overall and individual.
• Overall level: Balanced gradient direction aligns optimization across modalities, preventing suppression.
• Individual level: Aligns spatial structures of samples to mitigate alignment noise.

Main Results:

• Experiments on emotion recognition and intention understanding datasets show GSCon's effectiveness.
• The method successfully reduces optimization bias and interaction noise.
• Demonstrates improved performance in holistic emotional state inference.

Conclusions:

• GSCon offers a novel solution to challenges in multimodal emotion recognition.
• The approach ensures balanced optimization and effective inter-modal interaction.
• GSCon advances the field of emotion AI by improving holistic emotional state inference.