The Insight-Inference Loop: Efficient Text Classification via Natural Language Inference and Threshold-Tuning

Area of Science:

• Computational Social Science
• Natural Language Processing

Background:

• Social scientists face challenges in analyzing large text datasets due to manual annotation labor, technical complexity, and the gap between algorithms and theoretical understanding.
• Existing computational text classification methods require significant human-labeled data and machine learning expertise, limiting accessibility for many researchers.

Purpose of the Study:

• To propose a novel approach for large-scale text analysis that minimizes the need for human-labeled data and machine learning expertise.
• To efficiently integrate social scientists into the text analysis workflow, bridging the gap between computational methods and social science theory.
• To enable the detection of statements in text using advanced language models and active learning principles.

Main Methods:

• Utilized large language models pre-trained for natural language inference.
• Implemented a "few-shot" threshold-tuning algorithm based on active learning principles.
• Applied the approach to analyze tweets from the 2020 U.S. presidential election campaign.
• Benchmarked the proposed method against various computational approaches across three datasets.

Main Results:

• The proposed approach requires substantially less human-labeled data compared to traditional methods.
• The method demonstrates effectiveness in analyzing social media data, as shown by the 2020 U.S. election tweet analysis.
• The approach successfully integrates social scientists into the analytical workflow, facilitating theory-driven insights.

Conclusions:

• The developed text analysis approach lowers barriers to entry for social scientists, enabling broader use of computational methods.
• This method facilitates the extraction of meaningful insights from large text corpora by combining advanced AI with social science expertise.
• The approach offers a scalable and accessible solution for statement detection and analysis in social science research.