Jointly Aligning and Predicting Continuous Emotion Annotations.

Area of Science:

• Speech processing
• Affective computing
• Machine learning

Background:

• Dimensional emotion models (arousal, valence) capture dynamic emotional changes.
• Human reaction-time causes delays, desynchronizing speech and emotion labels.
• Accurate synchronization is crucial for analyzing emotion expression in speech.

Purpose of the Study:

• To introduce a novel convolutional neural network (multi-delay sinc network) for end-to-end speech emotion recognition.
• To address the challenge of label-speech desynchronization caused by inherent human reaction-time delays.
• To achieve state-of-the-art performance in predicting continuous emotion descriptors from speech.

Main Methods:

• Developed a multi-delay sinc network, a convolutional neural network architecture.
• Introduced a 'delayed sinc layer' implementing a time-shifted low-pass filter.
• Employed a gradient-based algorithm to learn single and non-stationary time-varying delays.
• Evaluated the system on the RECOLA and SEWA emotion datasets.

Main Results:

• The multi-delay sinc network achieved state-of-the-art speech-only results on emotion recognition tasks.
• The system successfully learned and compensated for time-varying delays between speech and emotion labels.
• Demonstrated the efficacy of the delayed sinc layer in aligning audio and label data.

Conclusions:

• The proposed multi-delay sinc network effectively synchronizes speech signals with continuous emotion labels.
• This approach advances speech-based emotion recognition by accurately predicting dimensional emotion descriptors.
• Learning time-varying delays is a key factor in improving the performance of emotion recognition systems.