DashFusion: Dual-Stream Alignment With Hierarchical Bottleneck Fusion for Multimodal Sentiment Analysis

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Computer Vision
• Speech Processing

Background:

• Multimodal sentiment analysis (MSA) integrates diverse data types for enhanced sentiment understanding.
• Current MSA methods face challenges in aligning and fusing features across modalities.
• Isolated approaches to alignment or fusion limit performance and efficiency.

Purpose of the Study:

• To propose a novel framework, DashFusion, addressing alignment and fusion challenges in MSA.
• To improve the accuracy and efficiency of sentiment analysis by integrating multimodal information.
• To establish a new benchmark for multimodal sentiment analysis performance.

Main Methods:

• Dual-stream alignment module for temporal and semantic synchronization using cross-modal attention and contrastive learning.
• Supervised contrastive learning (SCL) to refine modality features with label information.
• Hierarchical bottleneck fusion (HBF) for progressive integration of features via compressed bottleneck tokens.

Main Results:

• DashFusion achieved state-of-the-art (SOTA) performance on CMU-MOSI, CMU-MOSEI, and CH-SIMS datasets.
• Ablation studies validated the effectiveness of the proposed alignment and fusion techniques.
• The framework demonstrated a balance between high performance and computational efficiency.

Conclusions:

• DashFusion offers a significant advancement in multimodal sentiment analysis.
• The proposed alignment and fusion strategies are crucial for effective MSA.
• The framework provides a robust and efficient solution for complex sentiment analysis tasks.