Multimodal sentiment analysis based on multi-layer feature fusion and multi-task learning.

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Affective Computing

Background:

• Multimodal sentiment analysis (MSA) aims to predict human emotions using diverse data sources.
• Current MSA methods struggle with integrating unimodal features, handling inconsistent signals, and incomplete data.
• Traditional approaches fail to capture inter-modal dependencies, leading to suboptimal performance with asymmetric representations.

Purpose of the Study:

• To develop an advanced MSA model that addresses existing challenges in feature extraction and information fusion.
• To enhance the model's ability to extract robust unimodal features and effectively integrate multimodal information.
• To improve prediction stability and accuracy, especially when dealing with incomplete or conflicting data across modalities.

Main Methods:

• Proposed a unimodal feature extraction network (UFEN) for enhanced unimodal feature representation.
• Introduced a multi-task fusion network (MTFN) to improve inter-modal correlation and fusion.
• Employed multilayer feature extraction, attention mechanisms, and Transformer architectures to mine feature relationships.

Main Results:

• The proposed UFEN and MTFN model demonstrated superior performance on benchmark datasets (MOSI, MOSEI, SIMS).
• Achieved state-of-the-art results in multimodal sentiment analysis tasks.
• Indicated improved robustness in handling incomplete or inconsistent multimodal sentiment data.

Conclusions:

• The novel approach effectively addresses limitations in current multimodal sentiment analysis.
• The proposed UFEN and MTFN architecture offers a significant advancement in emotion recognition accuracy and reliability.
• The findings suggest a promising direction for future research in multimodal affective computing.