Cross-Modal Sentiment Sensing with Visual-Augmented Representation and Diverse Decision Fusion

Area of Science:

• Computer Science
• Artificial Intelligence
• Natural Language Processing

Background:

• Online media usage has increased, leading users to share experiences and opinions on social networks.
• Social data contains emotional and attitudinal information valuable for sentiment analysis.
• Existing sentiment analysis methods often focus solely on text or overlook intra-modal information in multimodal scenarios.

Purpose of the Study:

• To propose a hybrid fusion network for multimodal sentiment analysis.
• To effectively capture both inter-modal and intra-modal features.
• To enhance the accuracy and generalization capacity of sentiment analysis models.

Main Methods:

• A hybrid fusion network is developed, incorporating multi-head visual attention for feature extraction.
• Representation fusion extracts semantic and sentimental information from text guided by visual features.
• Decision fusion combines outputs from multiple base classifiers trained on diverse modal representations.
• A similarity loss function is used to improve model generalization by injecting decision diversity.

Main Results:

• The proposed hybrid fusion network demonstrates superior performance on five multimodal datasets.
• The model achieves higher accuracy in multimodal sentiment analysis tasks.
• The network exhibits improved generalization capacity compared to conventional approaches.

Conclusions:

• The hybrid fusion network effectively integrates inter-modal and intra-modal information for robust sentiment analysis.
• The proposed approach addresses limitations of existing methods by considering unique features within each modality.
• The model offers a promising direction for advancing multimodal sentiment analysis research and applications.