MemoCMT: multimodal emotion recognition using cross-modal transformer-based feature fusion.

Area of Science:

• Artificial Intelligence
• Speech Processing
• Machine Learning

Background:

• Transformer models excel in speech emotion recognition but are computationally expensive.
• Convolutional neural networks are faster but limited in capturing long-range speech patterns.
• Existing methods face challenges balancing accuracy and computational efficiency.

Purpose of the Study:

• To develop an efficient and accurate speech emotion recognition system.
• To introduce a novel cross-modal transformer (CMT) for analyzing local and global speech features with text.
• To leverage pre-trained models like HuBERT and BERT for feature extraction.

Main Methods:

• Proposed MemoCMT system utilizing a cross-modal transformer (CMT).
• Integration of HuBERT for audio feature extraction and BERT for text analysis.
• Application of various fusion techniques post-feature integration for emotion classification.

Main Results:

• MemoCMT achieved high performance on IEMOCAP and ESD benchmark corpora.
• The CMT with min aggregation yielded the highest unweighted accuracy (UW-Acc) of 81.33% and 91.93%.
• Weighted accuracy (W-Acc) reached 81.85% and 91.84% respectively.

Conclusions:

• MemoCMT demonstrates strong generalization capacity and robustness.
• The system offers an efficient solution for speech emotion recognition.
• Publicly available implementation promotes reproducibility and further research.