Multimodal GRU with directed pairwise cross-modal attention for sentiment analysis.

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Computer Vision

Background:

• Multimodal sentiment analysis integrates text, audio, and visual data to interpret human emotions.
• Existing methods struggle with asynchronous signals, long-term dependencies, and computational efficiency.
• Transformer models are powerful but often too resource-intensive for practical applications.

Purpose of the Study:

• To develop a computationally efficient and accurate model for multimodal sentiment analysis.
• To address challenges in synchronizing asynchronous signals and capturing inter-modal dependencies.
• To improve the performance of emotion recognition across various datasets.

Main Methods:

• Introduction of the multimodal GRU model (MulG).
• Utilizes a cross-modal attention mechanism for signal synchronization and dependency capture.
• Employs Gated Recurrent Unit (GRU) layers for efficient sequential data processing.

Main Results:

• MulG demonstrates superior performance over existing methods on CMU-MOSI, CMU-MOSEI, and IEMOCAP datasets.
• Achieved 82.2% accuracy on CMU-MOSI (7-class), 82.1% on CMU-MOSEI, and 90.6% on IEMOCAP.
• Ablation studies confirm the significant contribution of each model component.

Conclusions:

• MulG offers a practical and scalable solution for multimodal sentiment analysis.
• The model effectively handles asynchronous signals and long-term dependencies.
• MulG enhances applications in user-generated content analysis and human-computer interaction.