Multimodal Emotion Recognition on RAVDESS Dataset Using Transfer Learning.

Area of Science:

• Computer Science
• Artificial Intelligence
• Human-Computer Interaction

Background:

• Emotion recognition is crucial for applications in healthcare and road safety.
• Multimodal approaches combining speech and facial data offer potential for improved accuracy.

Purpose of the Study:

• To develop and evaluate a multimodal emotion recognition system using speech and facial information.
• To investigate the effectiveness of transfer learning techniques for speech emotion recognition.
• To propose a novel framework for facial emotion recognition and combine it with speech for enhanced performance.

Main Methods:

• Speech emotion recognition: Evaluated transfer learning (embedding extraction, Fine-Tuning) using CNN-14 (PANNs framework).
• Facial emotion recognition: Proposed a framework with a pre-trained Spatial Transformer Network and a bi-LSTM with attention.
• Multimodal fusion: Employed a late fusion strategy to combine speech and facial modalities.

Main Results:

• Fine-tuning CNN-14 yielded the most robust speech emotion recognition results.
• The facial emotion recognition framework showed promise, though frame-based systems require further research for video tasks.
• The combined multimodal system achieved 80.08% accuracy on the RAVDESS dataset for classifying eight emotions using subject-wise 5-fold cross-validation.

Conclusions:

• Speech and facial modalities contain significant information for detecting emotional states.
• Combining speech and facial recognition through late fusion effectively improves overall system performance.
• The findings highlight the potential of multimodal emotion recognition in real-world applications.