Sentiment Analysis and Emotion Recognition from Speech Using Universal Speech Representations

Area of Science:

• Speech processing
• Computational linguistics
• Machine learning

Background:

• Analyzing sentiment and emotion in human speech is complex, especially with only audio data from sources like phone calls.
• Self-supervised learning (SSL) offers a promising avenue for extracting meaningful representations from speech data.

Purpose of the Study:

• To evaluate the effectiveness of recent SSL models, specifically speaker-aware pre-trained universal speech representations, for sentiment analysis and emotion recognition from speech.
• To assess the performance of different model sizes across various sentiment and emotion tasks.

Main Methods:

• Utilized three sizes of universal speech representation models pre-trained with speaker-aware techniques.
• Independently evaluated model performance on three distinct sentiment analysis tasks and one emotion recognition task.
• Measured performance using weighted and unweighted accuracy scores.

Main Results:

• Achieved the highest accuracy in binary sentiment analysis, with weighted accuracy at 81% and unweighted accuracy at 73%.
• Binary classification using unimodal acoustic analysis demonstrated competitive performance against prior multimodal fusion methods.
• Models exhibited poor performance in emotion recognition and multi-class sentiment analysis tasks (3, 6, and 7 classes).

Conclusions:

• SSL models show strong potential for binary speech sentiment analysis, even with unimodal audio data.
• Performance limitations in emotion recognition and multi-class sentiment tasks may stem from dataset imbalances.
• Further research is needed to address challenges in complex speech analysis tasks and dataset properties.