Evaluating the statistical realism of LLM-generated social science data

Area of Science:

• Social Sciences
• Computational Social Science
• Artificial Intelligence

Background:

• Large language models (LLMs) offer potential for generating social science data.
• Previous research focused on individual-level data characteristics.
• A gap exists in evaluating population-level statistical validity of LLM-generated data.

Purpose of the Study:

• To propose a framework for assessing the validity of LLM-generated social science data.
• To introduce SSDataBench, a benchmark for evaluating population-level statistical realism.
• To identify limitations of current LLMs in reproducing real-world statistical patterns.

Main Methods:

• Developed SSDataBench to assess five key statistical patterns: univariate distributions, bivariate associations, multivariate outcome predictions, life event sequence distributions, and associations between life event sequences and covariates.
• Applied SSDataBench to four longitudinal and three cross-sectional datasets across six social domains (demographics, socioeconomic status, marriage, health, abilities, attitudes).
• Evaluated LLM-generated data against real-world population-level statistics.

Main Results:

• Current LLMs exhibit representational limitations, particularly under sparse conditioning.
• LLMs tend to compress real-world heterogeneity into simplified typological structures.
• Domain-specific training shows promise for enhancing population-level statistical realism.

Conclusions:

• Assessing LLM-generated data requires a focus on population-level statistical patterns, mirroring survey research principles.
• SSDataBench provides a systematic method for evaluating statistical realism in LLM outputs.
• Future work should focus on improving LLM architectures and training strategies to enhance statistical validity for social science research.