Self-adaptive attention fusion for multimodal aspect-based sentiment analysis.

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Computer Vision

Background:

• Multimodal aspect term extraction (MATE) and multimodal aspect-oriented sentiment classification (MASC) are key components of multimodal sentiment analysis.
• Pretrained generative models show promise for aspect-based sentiment analysis (ABSA), but face challenges in cross-modal transfer due to semantic gaps.
• Existing methods struggle to effectively integrate textual and visual information for sentiment analysis tasks.

Purpose of the Study:

• To propose a self-adaptive cross-modal attention fusion architecture for joint multimodal aspect-based sentiment analysis (JMABSA).
• To bridge the semantic gap between textual and visual modalities in generative models.
• To adapt textual-based pretraining models for multimodal sentiment analysis.

Main Methods:

• Developed a generative model employing an image-text selective fusion mechanism.
• Implemented a self-adaptive cross-modal attention fusion architecture.
• Utilized pretrained generative models adapted for multimodal input.

Main Results:

• The proposed JMABSA model significantly outperforms state-of-the-art approaches on two benchmark datasets.
• Demonstrated effective bridging of the semantic gap between text and image representations.
• Showcased successful adaptation of textual pretraining for multimodal sentiment analysis.

Conclusions:

• The novel architecture provides a significant advancement in joint multimodal aspect-based sentiment analysis.
• The selective fusion mechanism effectively addresses cross-modal semantic discrepancies.
• The model offers a robust solution for sentiment analysis in image-text data.