Adaptive multimodal transformer based on exchanging for multimodal sentiment analysis.

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Computer Vision

Background:

• Multimodal sentiment analysis leverages cross-modal emotional cues for enhanced classification.
• Existing methods struggle with modal distribution differences, inefficient cross-modal interaction, and contextual correlation modeling.

Purpose of the Study:

• To propose an Adaptive Multimodal Transformer based on Exchanging (AMTE) model to address challenges in multimodal sentiment analysis.
• To improve sentiment classification performance by efficiently fusing cross-modal emotional cues.

Main Methods:

• The proposed AMTE model utilizes an exchange fusion mechanism to enhance local features with global features from other modalities.
• A multi-scale hierarchical fusion mechanism creates an adaptive hyper-modal representation, reducing inter-modal distribution differences.
• The language modality acts as the dominant modality for deep fusion with the hyper-modal representation and contextual information.

Main Results:

• AMTE achieved high binary sentiment classification accuracies: 89.18% on CMU-MOSI, 88.28% on CMU-MOSEI, and 81.84% on CH-SIMS.
• The model effectively bridges cross-modal semantic differences while preserving modality specificity.
• Efficient fusion and reduced distribution differences contribute to superior performance.

Conclusions:

• The AMTE model demonstrates significant improvements in multimodal sentiment analysis.
• The proposed exchange fusion and hierarchical fusion mechanisms are effective in handling cross-modal challenges.
• AMTE offers a promising approach for accurate sentiment classification in multimodal data.