Fine-Tuning a Pretrained CNN Combined with Self-Attention for Speech Emotion Recognition

Speech emotion recognition (SER) technology has broad applications in healthcare, including enhancing patient monitoring, supporting mental health diagnosis, and improving human-computer interaction in assistive systems. Prior works in SER have combined convolutional neural networks (CNNs) with recurrent neural networks (RNNs) to model spatial and temporal dependencies. However, RNNs are computationally expensive and constrained by sequential processing. To address these challenges, this work integrates a self-attention mechanism with a CNN to model temporal relationships. Speech signals are divided into segments and transformed into Mel-spectrogram representations, which serve as input to a pretrained CNN. The CNN captures intra-spectrogram features, while the self-attention mechanism models inter-spectrogram dependencies, enabling the architecture to learn spatial and temporal emotional patterns. The proposed network is fine-tuned on an SER dataset comprising angry, happy, sad, and neutral emotions. It achieves an unweighted average recall of 0.812 in subject-independent cross-validation, demonstrating its ability to discern emotional states in unseen subjects. The integration of self-attention with CNN improves emotion recognition performance and highlights its potential in emotion-sensitive healthcare technologies and other domains where understanding human affect is important.