Rethinking Emotion Annotations in the Era of Large Language Models

Area of Science:

• Affective computing
• Natural Language Processing
• Human-Computer Interaction

Background:

• Affective computing systems require extensive human-annotated emotion datasets for training and evaluation.
• Human annotation is costly, subjective, and challenging to quality control.
• Large Language Models (LLMs) demonstrate strong performance in Natural Language Understanding tasks, suggesting potential for automated annotation.

Purpose of the Study:

• To analyze the complexities of emotion annotation using LLMs, specifically GPT-4.
• To evaluate GPT-4's performance in emotion perception compared to human annotators.
• To explore methods for integrating GPT-4 into emotion annotation pipelines to improve efficiency and quality.

Main Methods:

• Conducted experiments using GPT-4 for emotion annotation.
• Performed human evaluation studies to compare GPT-4's annotations with human labels.
• Investigated two integration strategies for LLMs in emotion annotation workflows.

Main Results:

• GPT-4 achieved high ratings in human evaluation, surpassing previous benchmarks where only human labels were ground truth.
• Observed discrepancies between human and GPT-4 emotion perception, highlighting the necessity of human oversight.
• Demonstrated GPT-4's potential to identify low-quality labels, decrease human annotator workload, and boost downstream model performance.

Conclusions:

• LLMs, particularly GPT-4, can significantly aid human annotators in emotion labeling tasks.
• A hybrid approach combining LLM capabilities with human judgment offers a promising direction for future emotion annotation practices.
• The findings suggest novel methods for emotion labeling, leveraging LLMs to enhance efficiency and accuracy in affective computing.