Utterance Level Feature Aggregation with Deep Metric Learning for Speech Emotion Recognition

Area of Science:

• Artificial Intelligence
• Speech Processing
• Machine Learning

Background:

• Speech conveys paralinguistic emotional information crucial for human-computer interaction.
• Automatic Speech Emotion Recognition (SER) is vital for applications like mental health diagnosis and human behavior analysis.
• Current SER methods struggle with robust utterance-level feature representation.

Purpose of the Study:

• To introduce a novel SER method overcoming limitations of state-of-the-art techniques.
• To enhance feature representation for robust, single-utterance vector aggregation.
• To improve latent space data representation through an emotionally constrained loss function.

Main Methods:

• Utilized a Squeeze and Excitation ResNet (SE-ResNet) model with spectrogram inputs.
• Extended the CNN architecture with a trainable discriminative GhostVLAD clustering layer for feature aggregation.
• Implemented an end-to-end neural embedding approach with an emotionally constrained triplet loss function.

Main Results:

• Achieved 83.35% accuracy on the RAVDESS dataset and 64.92% on the CREMA-D dataset.
• Demonstrated accuracy gains superior to 24% compared to human observers.
• Showcased accuracy improvements of over 3% against state-of-the-art methods.

Conclusions:

• The proposed SE-ResNet and GhostVLAD-based SER method offers robust utterance-level feature representation.
• The emotionally constrained triplet loss function effectively improves latent space representation.
• The methodology significantly advances the state-of-the-art in automatic speech emotion recognition.