Measuring complex constructs in large-scale text with computational social mixed methods

Area of Science:

• Computational Social Science
• Machine Learning
• Text Analysis

Background:

• Social science and machine learning are converging for large-scale text analysis.
• Current methods often separate conceptual validity and technical performance, reducing measurement quality.
• Bridging this gap is crucial for robust computational social science.

Purpose of the Study:

• To provide guidelines for computational social mixed methods pipelines.
• To address pitfalls in data annotation, model training, and statistical analysis.
• To improve the validity and comparability of computational social science.

Main Methods:

• Developed "Iterative Annotation" and "Training on Confident Examples" techniques.
• Integrated social science and machine learning expertise across three pipeline stages: annotation, training, and analysis.
• Evaluated large language models as alternatives to transformer-based classifiers.

Main Results:

• Demonstrated improved validity and comparability in a case study on online hate.
• Addressed common issues like unbalanced data and stagnant model performance.
• Showcased the benefits of a unified approach to computational social science.

Conclusions:

• Integrating social science and machine learning expertise enhances computational social science.
• Computational social mixed methods pipelines offer a framework for valid and comparable text analysis.
• The proposed methods improve measurement quality in analyzing complex social phenomena.