Text-in-Image Enhanced Self-Supervised Alignment Model for Aspect-Based Multimodal Sentiment Analysis on Social...

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Computer Vision

Background:

• Social media's growth necessitates advanced opinion mining and sentiment analysis on multimodal data.
• Aspect-based multimodal sentiment analysis (ABMSA) is crucial for fine-grained sentiment determination.
• Existing ABMSA methods struggle with social media data due to overlooked embedded image text.

Purpose of the Study:

• To develop a model that comprehensively analyzes multimodal information for ABMSA.
• To enhance sentiment analysis accuracy by integrating text extracted from images.
• To reduce noise interference and improve focus on relevant semantic features.

Main Methods:

• Proposed the Text-in-Image Enhanced Self-supervised Alignment Model (TESAM).
• Utilized Optical Character Recognition (OCR) to extract embedded text from images.
• Fused extracted text with visual features and incorporated aspect words for focused analysis.
• Employed self-supervised alignment pre-training to bridge the semantic gap between modalities using Euclidean distance and cosine similarity.

Main Results:

• TESAM demonstrated remarkable performance on three ABMSA benchmarks.
• The model effectively integrated visual and textual information, including embedded image text.
• Incorporating aspect words successfully reduced interference from irrelevant semantic features.

Conclusions:

• The proposed TESAM model significantly improves aspect-based multimodal sentiment analysis on social media data.
• Integrating embedded image text and aspect-guided feature selection enhances sentiment analysis accuracy.
• Self-supervised alignment is effective in mitigating the semantic gap between different modalities.