Research on cross-modal emotion recognition based on multi-layer semantic fusion

Area of Science:

• Artificial Intelligence
• Machine Learning
• Affective Computing

Background:

• Multimodal emotion analysis integrates diverse data for enhanced understanding.
• Existing methods face challenges with signal heterogeneity and noisy data.
• Advanced feature extraction and fusion are crucial for accurate emotion recognition.

Purpose of the Study:

• To propose the Cross-modal Emotion Recognition based on multi-layer semantic fusion (CM-MSF) model.
• To leverage inter-modal complementarity for adaptive feature extraction.
• To improve accuracy and reduce misjudgment in emotion classification.

Main Methods:

• A parallel deep learning module for cost-effective, in-depth feature extraction from individual modalities.
• A cascaded cross-modal encoder using Bidirectional Long Short-Term Memory (BILSTM) and Convolutional 1D (ConV1d) for inter-modal fusion.
• Mask-gated Fusion Networks (MGF-module) for adaptive information selection and flow control.

Main Results:

• The CM-MSF model achieved high performance on CMU-MOSI and CMU-MOSEI datasets.
• Binary classification accuracies reached 89.1% (CMU-MOSI) and 88.6% (CMU-MOSEI).
• F1 scores were 87.9% (CMU-MOSI) and 88.1% (CMU-MOSEI), demonstrating effectiveness.

Conclusions:

• The CM-MSF model effectively integrates multimodal information for superior emotion recognition.
• The proposed fusion strategy addresses signal heterogeneity and redundant information challenges.
• Experimental results validate the model's accuracy and robustness in classifying emotions.