Open-source LLMs for text annotation: a practical guide for model setting and fine-tuning

Area of Science:

• Political Science
• Computational Social Science
• Natural Language Processing

Background:

• Open-source Large Language Models (LLMs) are increasingly used for text analysis.
• Scholars require guidance on LLM performance for specific political science tasks.
• Establishing benchmarks for LLM effectiveness in social science research is crucial.

Purpose of the Study:

• To evaluate the performance of open-source LLMs in political science text classification.
• To compare zero-shot and fine-tuned LLM capabilities on tasks like stance, topic, and relevance.
• To provide a benchmark for LLM effectiveness and inform scholarly decision-making.

Main Methods:

• Assessed open-source LLMs using both zero-shot and fine-tuned approaches.
• Utilized news articles and tweets datasets for text annotation tasks.
• Compared fine-tuning against few-shot training with limited annotated data.

Main Results:

• Fine-tuning enhances open-source LLM performance, matching or exceeding zero-shot GPT-3.5 and GPT-4.
• Fine-tuned open-source LLMs still lag behind fine-tuned GPT-3.5.
• Fine-tuning is more effective than few-shot training with modest data.

Conclusions:

• Fine-tuned open-source LLMs are suitable for diverse text annotation applications in political science.
• The study provides a practical benchmark for LLM performance in social science research.
• A Python notebook is available to assist researchers in applying LLMs for text annotation.