Speech emotion recognition using fine-tuned Wav2vec2.0 and neural controlled differential equations classifier

Area of Science:

• Artificial Intelligence
• Machine Learning
• Signal Processing

Background:

• Speech emotion recognition (SER) is crucial for applications like social media and medical diagnostics.
• Existing SER datasets often suffer from small volumes and high complexity, posing modeling challenges.
• Effective integration and modeling of audio data remain significant hurdles in SER research.

Purpose of the Study:

• To propose a novel model architecture for speech emotion recognition.
• To address the challenges of small data volumes and high complexity in SER datasets.
• To improve the accuracy and efficiency of SER models.

Main Methods:

• A hybrid model architecture combining fine-tuned Wav2vec2.0 and Neural Controlled Differential Equations (NCDEs).
• Wav2vec2.0 is utilized for extracting rich contextual audio features.
• NCDEs, with an MLP as the vector field, are employed to model high-dimensional time series features for classification.

Main Results:

• The model achieved a weighted accuracy of 73.37% and an unweighted accuracy of 74.18% on the IEMOCAP dataset.
• The proposed model demonstrated rapid convergence, achieving good accuracy within a single training epoch.
• Exceptional model stability was observed, with low standard deviations for both weighted (0.45%) and unweighted (0.39%) accuracy.

Conclusions:

• The proposed Wav2vec2.0 and NCDEs model offers an effective solution for speech emotion recognition.
• The model's rapid convergence and high stability make it suitable for practical SER applications.
• This approach advances SER by efficiently modeling complex audio data characteristics.