Audio-Based Emotion Recognition Using Self-Supervised Learning on an Engineered Feature Space

Area of Science:

• Affective computing
• Machine learning
• Speech processing

Background:

• Emotion recognition from audio is crucial for interactive systems in various fields.
• A key challenge is the limited availability of labeled training data for high-performance models.
• Self-supervised learning (SSL) offers a solution by learning from data properties without extensive labels.

Purpose of the Study:

• To investigate the effectiveness of self-supervised learning pre-training for audio-based emotion recognition.
• To apply SSL to encoded acoustic features from the CMU-MOSEI dataset.
• To evaluate the impact of SSL on model performance compared to a baseline deep learning model.

Main Methods:

• Applied self-supervised learning pre-training to encoded acoustic data (74 features) from the CMU-MOSEI dataset.
• Pre-trained the model to predict masked acoustic data timestamps.
• Fine-tuned the pre-trained model using a small set of annotated data.
• Evaluated performance using Mean Absolute Error (MAE) and four-class accuracy, comparing against a baseline.

Main Results:

• Self-supervised learning consistently improved model performance across all evaluated metrics (MAE, accuracy).
• Performance gains were most significant when the amount of annotated data for fine-tuning was small.
• SSL demonstrated notable improvements for easily classifiable emotions like happy, sad, and angry.
• SSL improved performance even when applied to embedded feature representations, not just raw audio data.

Conclusions:

• Self-supervised learning is highly beneficial for audio-based emotion recognition, particularly in low-data regimes.
• SSL effectively enhances affective computing models by leveraging unlabeled data.
• The study validates SSL's utility on encoded acoustic features, offering a practical approach for improving emotion recognition systems.