Combining Electrodermal Activity and Speech Analysis towards a more Accurate Emotion Recognition System

Area of Science:

• Affective computing
• Human-computer interaction
• Psychophysiology

Background:

• Emotion recognition research integrates physiological, behavioral, and speech data.
• Electrodermal activity (EDA) is a key psychophysiological arousal indicator, but challenging in real-world scenarios.
• Speech contains valuable emotional state information, yet its potential in affective computing is underexplored.

Purpose of the Study:

• To investigate merging electrodermal activity (EDA) and speech data for enhanced human arousal level recognition.
• To focus on recognizing the speaker's arousal state, distinct from the emotion conveyed by the spoken word.
• To develop a robust method for analyzing arousal during single affective word pronunciation.

Main Methods:

• Utilized a support vector machine with recursive feature elimination (SVM-RFE) strategy.
• Trained and tested models on three datasets: speech only, EDA only, and combined speech-EDA.
• Focused on speaker arousal rather than the semantic content of spoken words.

Main Results:

• The fusion of EDA and speech data significantly improved arousal recognition by +11.64% compared to individual channels.
• Recursive feature elimination identified six key features for future multivariate emotion modeling.
• The combined approach demonstrated superior performance in distinguishing arousal levels.

Conclusions:

• Merging physiological (EDA) and speech signals offers a significant advantage for accurate arousal recognition.
• This multimodal approach provides a more comprehensive understanding of a speaker's internal state.
• The identified features pave the way for advanced, multivariate emotion recognition models.