Algorithmic bias in social research: A meta-analysis

Area of Science:

• Social sciences, including management, political science, and sociology.
• Interdisciplinary research methodologies.

Background:

• A significant "reproducibility crisis" is impacting both natural and social sciences.
• Selective reporting and methodological issues are key contributing factors to this crisis.

Purpose of the Study:

• To investigate the fusion of selective reporting and methodological problems.
• To demonstrate how the uncritical adoption of Boolean optimization algorithms has induced "algorithmic bias" in social science research.
• To assess the scale of this bias in empirical studies using Qualitative Comparative Analysis (QCA).

Main Methods:

• Analysis of replication material from 215 peer-reviewed QCA articles.
• Examination of studies published in 109 high-profile management, political science, and sociology journals.
• Estimation of the prevalence and severity of algorithmic bias in empirical QCA work.

Main Results:

• An estimated one in three studies employing QCA is affected by algorithmic bias.
• Approximately one in ten of these studies are severely impacted by the bias.
• The findings highlight a considerable scale of algorithmic bias over the last 25 years.

Conclusions:

• The uncritical import of algorithms like Boolean optimization can introduce significant bias across disciplines.
• Scientists must rigorously evaluate the suitability of methods and algorithms before cross-disciplinary application.
• Addressing algorithmic bias is crucial for mitigating the reproducibility crisis in social sciences.