Does big data serve policy? Not without context. An experiment with in silico social science

Area of Science:

• Computational Social Science
• In Silico Social Science
• Quantitative Social Science Methodology

Background:

• DARPA Ground Truth project evaluated social science through simulated worlds with hidden causality.
• Scientists aimed to collect data, discover causal structures, predict futures, and prescribe policies within these simulated environments.
• The experiment's ground truth was known to the system but not to the participating scientists.

Purpose of the Study:

• To evaluate the limits of contemporary quantitative social science methodology.
• To identify challenges in problem-solving, data interpretation, and model relevance within simulated social worlds.
• To explore how computational social science can be adapted for policy-oriented applications.

Main Methods:

• Construction of four varied simulated social worlds with hidden causality.
• Deployment of scientist teams to collect data and analyze simulated social dynamics.
• Exploration of diverse quantitative methods including probabilistic programming, deep neural networks, and predictive probabilistic finite state machines.
• Application of competing approaches by distinct subteams and the broader TopCoder.com community.

Main Results:

• Problem-solving without a shared ontology significantly limits quantitative analysis.
• Data labels can bias analyst associations, affecting the interpretation of simulated causal processes.
• Focusing solely on novel causes limits the practical relevance of models for policy and problem-solving.
• Singular quantitative methods were often insufficient; a pluralistic approach enabled discovery of underlying structures.

Conclusions:

• Imperfect knowledge can suffice for robust prediction when a pluralistic approach is adopted.
• Policy-oriented computational social science requires greater tolerance for failure, diversity, uncertainty, and complexity.
• Current computational social science practices need adaptation to better serve policy needs.