AVaTER: Fusing Audio, Visual, and Textual Modalities Using Cross-Modal Attention for Emotion Recognition

Area of Science:

• Computer Science
• Artificial Intelligence
• Natural Language Processing

Background:

• Multimodal emotion classification (MEC) integrates audio, video, and text for robust emotion recognition.
• Challenges include fusing diverse data modalities and the lack of Bangla-specific datasets.
• Existing systems struggle with nuanced emotional expression in Bangla.

Purpose of the Study:

• To introduce the MAViT-Bangla dataset, a novel multimodal resource for Bangla emotion recognition.
• To develop and evaluate a cross-modal attention framework (AVaTER) for enhanced MEC.
• To address the limitations of unimodal approaches in Bangla emotion analysis.

Main Methods:

• Created MAViT-Bangla dataset with 1002 audio, video, and text samples covering anger, fear, joy, and sadness.
• Developed the AVaTER framework utilizing cross-modal attention for feature fusion.
• Evaluated the framework's performance against unimodal methods.

Main Results:

• The MAViT-Bangla dataset provides a comprehensive resource for Bangla MEC research.
• The AVaTER framework achieved an F1-score of 0.64.
• This represents a significant improvement over unimodal emotion recognition techniques.

Conclusions:

• The MAViT-Bangla dataset is a valuable contribution to multimodal emotion recognition research in Bangla.
• The AVaTER framework effectively integrates multimodal features for improved emotion classification accuracy.
• Future work can leverage this dataset and framework for more sophisticated Bangla emotion understanding.