Multimodal Sentiment Analysis Representations Learning via Contrastive Learning with Condense Attention Fusion

Area of Science:

• Artificial Intelligence
• Machine Learning
• Natural Language Processing

Background:

• Multimodal sentiment analysis aims to predict user emotions by integrating diverse data sources.
• Effective data fusion and redundancy reduction are critical challenges in multimodal sentiment analysis.
• Existing methods struggle to optimally combine modalities and eliminate irrelevant information.

Purpose of the Study:

• To propose a novel multimodal sentiment analysis model addressing data fusion and redundancy challenges.
• To enhance data representation and extract richer multimodal features for improved sentiment prediction.
• To leverage supervised contrastive learning for more effective learning of sentiment-related features.

Main Methods:

• Developed a multimodal sentiment analysis model incorporating a novel MLFC module.
• Utilized Convolutional Neural Networks (CNN) and Transformers within the MLFC module to address feature redundancy.
• Employed supervised contrastive learning to improve the model's ability to learn standard sentiment features.

Main Results:

• The proposed model demonstrated superior performance compared to state-of-the-art methods on MVSA-single, MVSA-multiple, and HFM datasets.
• The MLFC module effectively reduced redundant information and irrelevant features from multiple modalities.
• Ablation experiments confirmed the efficacy of the proposed supervised contrastive learning approach and MLFC module.

Conclusions:

• The developed multimodal sentiment analysis model offers a significant advancement in data fusion and feature representation.
• Supervised contrastive learning combined with the MLFC module provides a robust framework for accurate sentiment analysis.
• The findings suggest a promising direction for future research in comprehensive multimodal emotion recognition.