Tri-MCA fusion: cross-modal attention and dynamic gating for multimodal sentiment analysis

Area of Science:

• Artificial Intelligence
• Machine Learning
• Natural Language Processing

Background:

• Multimodal sentiment analysis leverages text, audio, and visual data for emotion recognition.
• Current deep learning models struggle with heterogeneous modality interactions, imbalance, and noise.
• Existing fusion strategies often use static or limited cross-modal interactions, hindering performance.

Purpose of the Study:

• To propose an advanced architecture for multimodal sentiment analysis.
• To enhance the modeling of interactions among textual, acoustic, and visual modalities.
• To address challenges of modality imbalance and noisy signals in emotion recognition.

Main Methods:

• Developed a tri-modal cross-attention architecture with adaptive gating mechanisms.
• Introduced a Tri-modal cross-attention module for improved inter-modal interaction.
• Evaluated the framework on CMU-MOSI, CMU-MOSEI, and SIMS benchmark datasets.

Main Results:

• The proposed method achieved superior performance across multiple evaluation metrics.
• Achieved 85.6% Acc2 and 85.2 F1-score on CMU-MOSI, outperforming existing models.
• Demonstrated significant improvements on CMU-MOSEI and SIMS datasets.

Conclusions:

• The tri-modal cross-attention architecture with adaptive gating enhances multimodal sentiment analysis.
• Effective modeling of cross-modal dependencies is crucial for robust emotion prediction.
• The proposed approach offers a significant advancement in multimodal emotion recognition.