Multimodal transformer augmented fusion for speech emotion recognition.

Area of Science:

• Artificial Intelligence
• Human-Computer Interaction
• Signal Processing

Background:

• Speech emotion recognition (SER) is complex due to subjective and ambiguous emotional expression.
• Multimodal SER methods show promise but struggle with integrating heterogeneous data.
• Existing fusion techniques often overlook fine-grained interactions between modalities.

Purpose of the Study:

• To develop an advanced fusion strategy for improved multimodal speech emotion recognition.
• To address the challenge of effectively integrating information from different modalities.
• To capture fine-grained interactions within and between speech and text modalities.

Main Methods:

• Proposed a multimodal transformer augmented fusion method utilizing a hybrid fusion strategy.
• Implemented a Model-fusion module with Cross-Transformer Encoders for multimodal emotional representation.
• Employed feature-level and model-level fusion for enhanced speech feature extraction using text and multimodal features.

Main Results:

• The proposed method demonstrated superior performance compared to state-of-the-art approaches.
• Achieved significant improvements on the IEMOCAP and MELD datasets.
• Successfully captured fine-grained modal interactions for more accurate emotion recognition.

Conclusions:

• The multimodal transformer augmented fusion method offers a breakthrough in integrating heterogeneous multimodal data for SER.
• The hybrid fusion strategy and Cross-Transformer Encoders effectively enhance multimodal emotional representation.
• This approach advances the field of speech emotion recognition by enabling more nuanced understanding of emotional cues.