Are chatbots reliable text annotators? Sometimes.

Area of Science:

• Social Sciences
• Computational Linguistics
• Artificial Intelligence

Background:

• ChatGPT shows promise for social science text annotation but has drawbacks (closed-source, transparency, reproducibility, cost, data protection).
• Open-source (OS) large language models (LLMs) offer a potential alternative addressing these limitations.

Purpose of the Study:

• To systematically compare the performance of OS LLMs against ChatGPT and traditional supervised machine learning classifiers for text annotation tasks.
• To evaluate the impact of zero-shot, few-shot learning, and prompt variations on model performance.

Main Methods:

• Comparative evaluation of multiple OS LLMs and ChatGPT.
• Utilized zero-shot and few-shot learning with generic and custom prompts.
• Tested models on a new dataset of tweets from US news media for binary text annotation.
• Compared LLM performance against a supervised classification model (DistilBERT).

Main Results:

• Significant variation in performance was observed among ChatGPT and various OS LLMs across different annotation tasks.
• The supervised classifier using DistilBERT generally outperformed both ChatGPT and the evaluated OS LLMs.
• ChatGPT's performance was found to be unreliable for substantive text annotation.

Conclusions:

• Caution is advised when using ChatGPT for substantial text annotation in social science research due to performance variability and Open Science challenges.
• OS LLMs present an alternative but require careful evaluation; supervised methods like DistilBERT remain a strong benchmark.
• Further research is needed to optimize OS LLMs for social science applications and ensure transparency and reproducibility.