A systematic evaluation of text mining methods for short texts: Mapping individuals' internal states from online...

Area of Science:

• Social and Behavioral Sciences
• Computational Social Science
• Natural Language Processing

Background:

• Online texts offer rich data on individual internal states.
• Manual coding by human experts is reliable but limits data scale.
• Automated text analysis methods are needed for large-scale textual data.

Purpose of the Study:

• Evaluate automatic text analysis methods for social science.
• Compare performance against trained human coders across four coding tasks.
• Identify optimal methods for analyzing internal states in large text datasets.

Main Methods:

• Compared dictionary-based methods, trained large language models (LLMs), and zero-shot classification with GPT-4.
• Evaluated performance on coding expressions of motives, norms, emotions, and stances.
• Assessed accuracy and false positive rates of different text analysis approaches.

Main Results:

• LLMs trained on manually coded data achieved the highest performance across all tasks.
• Dictionary methods frequently generated false positives despite identifying infrequent categories.
• GPT-4 zero-shot classification showed promise but underperformed trained LLMs.

Conclusions:

• Trained LLMs are the most effective method for large-scale analysis of internal states in online texts.
• Simpler methods can be effective in specific contexts, offering a trade-off between complexity and performance.
• Social scientists should consider model performance and application-specific needs when selecting text analysis tools.