Fine-Tuning a Pretrained CNN Combined with Self-Attention for Speech Emotion Recognition

Area of Science:

• Artificial Intelligence
• Machine Learning
• Signal Processing

Background:

• Speech emotion recognition (SER) is vital for healthcare applications like patient monitoring and mental health.
• Previous SER models often use CNNs and RNNs, but RNNs are computationally intensive and slow.
• A more efficient approach is needed to capture both spatial and temporal features in speech signals.

Purpose of the Study:

• To develop an improved SER system by integrating a self-attention mechanism with a CNN.
• To overcome the limitations of RNNs in processing sequential data for emotion recognition.
• To enhance the accuracy and efficiency of SER for healthcare and human-computer interaction.

Main Methods:

• Speech signals were segmented and converted into Mel-spectrograms.
• A pretrained CNN extracted intra-spectrogram features.
• A self-attention mechanism modeled inter-spectrogram dependencies to capture temporal patterns.
• The network was fine-tuned on a dataset with angry, happy, sad, and neutral emotions.

Main Results:

• The proposed CNN and self-attention model achieved an unweighted average recall of 0.812.
• Performance was validated using subject-independent cross-validation.
• The system demonstrated effectiveness in recognizing emotional states in unseen subjects.

Conclusions:

• Integrating self-attention with CNNs enhances SER performance.
• This approach offers a computationally efficient alternative to RNNs for SER.
• The developed SER technology holds significant potential for emotion-sensitive healthcare and other AI applications.