Comparing Manual and Machine Annotations of Emotions in Non-acted Speech

Area of Science:

• Psychology
• Computer Science
• Speech Processing

Background:

• Workplace psychological well-being necessitates accurate emotion detection.
• Speech is a non-obtrusive method for emotion capture, but requires robust annotation for non-acted corpora.
• Existing emotion detection methods struggle with nuanced, real-world speech data.

Purpose of the Study:

• To evaluate human perception of emotions in non-acted speech after a time lag.
• To assess inter-annotator agreement on perceived emotions and their intensity.
• To improve machine learning classification accuracy for emotion detection in speech.

Main Methods:

• Collected and annotated a non-acted speech database, including self-perception and third-party perception after a six-month gap.
• Analyzed inter-annotator agreement rates for different emotion categories (neutral, positive, negative).
• Trained and tested machine learning models on speech samples with high inter-annotator agreement for speaker-dependent and independent classification.

Main Results:

• Human annotators showed higher agreement on neutral (84%) and negative (74%) emotions compared to positive (38%) emotions.
• Machine learning achieved 80% classification accuracy on agreed-upon samples, a 7% improvement for speaker-dependent tasks.
• Silently expressed positive and negative emotions were frequently misclassified as neutral.
• Speaker-independent classification achieved 61% overall accuracy.

Conclusions:

• High-level human perception of emotion in speech correlates with low-level acoustic features.
• Improved annotation strategies, focusing on high-agreement samples, enhance machine learning performance.
• Further research is needed to address the misclassification of subtly expressed emotions.