Multimodal Sentiment Analysis Based on Dynamic Language Enhancement and Synergistic Cross-Modal Transformer

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Computer Vision
• Speech Processing

Background:

• Multimodal sentiment analysis (MSA) integrates verbal, visual, and acoustic data to understand human emotions.
• Existing MSA methods struggle with leveraging linguistic knowledge and managing data redundancy and heterogeneity across modalities.
• Challenges in multimodal fusion include intermodal heterogeneity and spurious cross-modal interactions.

Purpose of the Study:

• To propose a novel MSA approach, LESCT, that addresses limitations in current methods.
• To enhance the utilization of linguistic information and improve fusion strategies for multimodal data.
• To mitigate intramodal redundancy and noise interference while fostering effective intermodal and intramodal interactions.

Main Methods:

• Developed a dynamic language enhancement network (LEN) for feature extraction.
• Implemented a guided attention mechanism within LEN to capture contextual cues from language representations.
• Introduced a synergistic cross-modal Transformer (SCT) with a bimodal generator for multimodal fusion, employing a local-to-composite strategy.

Main Results:

• Achieved high accuracy on benchmark datasets: 86.43% on CMU-MOSI, 86.38% on CMU-MOSEI, and 81.35% on CH-SIMS.
• Demonstrated superior performance compared to state-of-the-art (SOTA) methods in multimodal sentiment analysis.
• The proposed LESCT effectively mitigates intramodal redundancy and noise, enhancing feature representation.

Conclusions:

• The LESCT approach significantly improves multimodal sentiment analysis accuracy.
• Dynamic linguistic enhancement and synergistic cross-modal fusion are effective strategies for MSA.
• The proposed method offers a robust solution for complex multimodal data challenges.