Sequential translation-based multimodal sentiment analysis under uncertain missing modalities.

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Computer Vision

Background:

• Multimodal Sentiment Analysis (MSA) integrates data from various sources for accurate sentiment classification.
• Missing modalities present a significant challenge in MSA, with existing methods struggling to reconstruct cross-modal semantics effectively.
• Current approaches often neglect text modality's importance and involve high model complexity.

Purpose of the Study:

• To propose a novel Sequential Translation-based Multimodal Sentiment Analysis (STMSA) model.
• To address the challenge of uncertain missing modalities in MSA.
• To improve the accuracy and efficiency of sentiment classification by leveraging text-centric cross-modal interactions.

Main Methods:

• Developed a text-centric bidirectional translation mechanism to map text with audio and video modalities.
• Implemented a low-complexity non-modal completion architecture using an encoder-decoder for joint representation fitting.
• Utilized semantic guidance from text to enhance cross-modal representation alignment.

Main Results:

• The proposed STMSA model demonstrated superior performance compared to 10 state-of-the-art baseline models.
• Achieved more accurate cross-modal representations by effectively exploring inter-modal connections.
• Validated the model's effectiveness on two public datasets: CMU-MOSI and IEMOCAP.

Conclusions:

• The STMSA model effectively tackles the issue of missing modalities in sentiment analysis.
• The text-centric approach enhances the fusion of information across modalities, leading to improved sentiment classification.
• The model offers a more efficient and accurate solution for multimodal sentiment analysis tasks.