Multimodal Affective Analysis Using Hierarchical Attention Strategy with Word-Level Alignment.

Area of Science:

• Affective computing
• Human-computer interaction
• Machine learning

Background:

• Multimodal affective computing faces challenges in feature extraction from heterogeneous data.
• Current fusion strategies often neglect time-dependent interactions between modalities.
• Accurate interpretation of human affect from multiple data sources remains difficult.

Purpose of the Study:

• To develop an advanced multimodal architecture for improved affect recognition.
• To address limitations in feature representation and fusion strategies in affective computing.
• To enhance the classification of utterance-level sentiment and emotion using text and audio data.

Main Methods:

• Introduced a hierarchical multimodal architecture incorporating attention mechanisms.
• Implemented word-level fusion to capture fine-grained interactions between text and audio.
• Utilized synchronized attention over modalities for enhanced feature integration.

Main Results:

• The proposed model significantly outperforms existing state-of-the-art approaches on benchmark datasets.
• Demonstrated superior performance in classifying sentiment and emotion from multimodal inputs.
• Achieved visual interpretability through synchronized attention mechanisms.

Conclusions:

• The developed hierarchical multimodal architecture effectively overcomes key challenges in affective computing.
• The model's fusion strategy provides a more nuanced understanding of human affect.
• Synchronized attention offers a promising direction for interpretable and accurate multimodal emotion recognition.