Text-image semantic relevance identification for aspect-based multimodal sentiment analysis.

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Computer Vision

Background:

• Aspect-based multimodal sentiment analysis (ABMSA) identifies sentiment towards specific aspects in multimodal data.
• Existing ABMSA models often fail to account for semantic irrelevance between text and image components.
• Multimodal concatenation in current models can be suboptimal when text and image lack coherence.

Purpose of the Study:

• To propose a novel Text-Image Semantic Relevance Identification (TISRI) model for ABMSA.
• To enhance the accuracy of sentiment analysis in multimodal datasets with potentially irrelevant text-image pairs.
• To improve the robustness of ABMSA models by dynamically managing image information based on semantic relevance.

Main Methods:

• Developed a multimodal feature relevance identification module to assess text-image semantic similarity.
• Implemented an image gate mechanism to dynamically control image information input.
• Integrated an attention mechanism for text-aware image representation during multimodal fusion.
• Utilized auxiliary image information to bolster visual feature representation.

Main Results:

• The TISRI model demonstrated competitive performance on two ABMSA Twitter datasets.
• Experimental results validated the effectiveness of the proposed semantic relevance identification and dynamic information control methods.
• The attention-based fusion effectively prevented irrelevant image information from interfering with sentiment analysis.

Conclusions:

• The TISRI model offers a significant advancement in aspect-based multimodal sentiment analysis.
• Dynamically controlling image information based on semantic relevance is crucial for improving ABMSA performance.
• The proposed approach effectively handles multimodal data where text and image may not be semantically aligned.