GPT-4 as an X data annotator: Unraveling its performance on a stance classification task

Area of Science:

• Natural Language Processing (NLP)
• Artificial Intelligence (AI)

Background:

• Data annotation for NLP is expensive and time-consuming, especially for social media text due to its unique characteristics.
• Human annotators require extensive training and exhibit varying perceptions, impacting labeling consistency.

Purpose of the Study:

• To establish a performance baseline for GPT-4 as a social media text annotator for stance detection.
• To compare GPT-4's performance using Zero-shot, Few-shot, and Zero-shot with Chain-of-Thoughts prompting techniques.

Main Methods:

• Utilized a custom-labeled tweet dataset for stance detection.
• Experimented with three GPT-4 prompting techniques: Zero-shot, Few-shot, and Zero-shot with Chain-of-Thoughts.
• Fine-tuned transformer and traditional machine learning models on various label sets, including human labels, for comparison.

Main Results:

• GPT-4 achieved comparable results with Few-shot and Zero-shot Chain-of-Thoughts prompting.
• No GPT-4 prompting technique surpassed models fine-tuned on human labels.
• Zero-shot Chain-of-Thoughts proved effective for aspect-based social media text labeling.

Conclusions:

• GPT-4 shows potential as a social media text annotator, particularly with advanced prompting strategies.
• Human-labeled data remains superior for training high-performance stance detection models.
• Zero-shot Chain-of-Thoughts offers a promising, efficient alternative for specific social media annotation tasks.