Multimodal emotion recognition via adaptive high-order transforme network.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Affective Computing

Background:

• Multimodal emotion recognition combines various data types for improved accuracy.
• Current methods face challenges with data redundancy and feature discriminability.
• Advanced feature learning is crucial for robust emotion recognition.

Purpose of the Study:

• To propose a novel Adaptive High-order Transformer Network (AHOT) for enhanced multimodal emotion recognition.
• To address information redundancy and improve feature co-learning across modalities.
• To develop a method for learning highly discriminative high-order features.

Main Methods:

• Developed Adaptive Selection Transformer (AST) blocks for modality-specific feature extraction.
• Implemented Cross-modal Feature Fusion (CMFF) blocks to capture inter-modal interactions.
• Introduced a sparse high-order feature learning module for discriminative representations.

Main Results:

• The proposed AHOT method demonstrated superior emotion recognition accuracy on IEMOCAP and CMU-MOSEI datasets.
• AHOT outperformed several existing related methods in experimental evaluations.
• Ablation studies confirmed the effectiveness of individual components within AHOT.

Conclusions:

• The AHOT model effectively improves multimodal emotion recognition performance.
• The proposed network architecture successfully captures non-redundant features and inter-modal dynamics.
• AHOT offers a promising approach for advanced affective computing applications.