Coding with the machines: machine-assisted coding of rare event data.

Area of Science:

• Political Science
• Computational Social Science
• Artificial Intelligence

Background:

• Machine coding using LLMs has advanced significantly, but concerns exist regarding the reliability and validation of LLM classifications.
• LLM performance varies based on prompts, tuning, subject areas, tasks, and especially in zero-shot applications.

Purpose of the Study:

• To evaluate the performance of supervised and semi-supervised machine coding algorithms in political science.
• To compare the performance of three LLM models against each other and against trained human experts.
• To assess the impact of prompt engineering and data pre-processing on LLM coding accuracy.

Main Methods:

• Comparative analysis of three LLM models using supervised and semi-supervised learning on political data.
• Multiple iterations of model performance testing with varying prompt engineering and data pre-processing techniques.
• Assessment of LLM performance on an updated dataset to mitigate pre-training bias concerns.

Main Results:

• GPT-4 demonstrated performance comparable to trained human experts in coding familiar contexts.
• LLM consistency in coding varied across different contexts, with GPT-4 showing higher consistency.
• Prompt engineering and data pre-processing influenced LLM performance, but expert-level coding was primarily achieved by GPT-4 in specific scenarios.

Conclusions:

• Only GPT-4 approaches the performance of trained expert coders for political data, particularly in familiar contexts.
• LLM coding offers potential benefits but requires careful validation and consideration of drawbacks for reliable application.
• Further research is needed to refine LLM validation methods and ensure consistent performance across diverse coding tasks.