Dynamic Invariant-Specific Representation Fusion Network for Multimodal Sentiment Analysis.

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Computer Vision

Background:

• Multimodal sentiment analysis (MSA) integrates linguistic, auditory, and visual data to infer emotions.
• Existing challenges in MSA include effectively capturing heterogeneous data interactions.
• Advanced representation and fusion techniques are crucial for improving MSA performance.

Purpose of the Study:

• To address the limitations in heterogeneous data interaction for MSA.
• To propose a novel framework, the dynamic invariant-specific representation fusion network (DISRFN), for enhanced MSA.
• To validate the effectiveness of the proposed DISRFN framework on benchmark datasets.

Main Methods:

• Utilized an improved joint domain separation network to obtain joint domain separation representations from all modalities, maximizing redundant information.
• Employed a hierarchical graph fusion net (HGFN) for dynamic fusion of representations, enabling better multimodal data interaction.
• Conducted comparative experiments on MOSI and MOSEI datasets, including ablation studies on fusion strategy and loss functions.

Main Results:

• The DISRFN framework demonstrated superior performance in multimodal sentiment analysis tasks.
• Experimental results confirmed the effectiveness of the proposed joint domain separation and hierarchical graph fusion approaches.
• Ablation studies validated the contribution of individual components and the chosen loss functions.

Conclusions:

• The DISRFN framework offers a robust solution for overcoming challenges in multimodal data interaction for sentiment analysis.
• The proposed method significantly advances the state-of-the-art in multimodal sentiment analysis.
• The study highlights the importance of effective representation and fusion strategies in MSA.