The Bots Ruining Social Science Are Not Bots at All

Area of Science:

• Behavioral Science
• Human-Computer Interaction
• Data Science

Background:

• Online data collection is crucial for behavioral science research.
• Low-quality data, often attributed to bots, threatens the reliability of online studies.
• The actual source of problematic online data remains under-examined.

Purpose of the Study:

• To investigate the primary sources of data-quality issues in online behavioral research.
• To differentiate between bot-generated and human-generated problematic data.
• To inform strategies for enhancing data integrity in online studies.

Main Methods:

• Analysis of data from popular participant recruitment platforms like Amazon Mechanical Turk (MTurk) and Lucid.
• Multi-method approach across four studies conducted over five years.
• Identification of behavioral patterns indicative of fraudulent human participants.

Main Results:

• Evidence suggests fraudulent human users, primarily from outside the United States, are the main source of poor-quality data, not bots.
• Telltale signs of fraudulent human responses were identified.
• Effective methods for blocking problematic human responses were described.

Conclusions:

• The assumption that bots cause most online data-quality problems is likely incorrect.
• Targeting fraudulent human users is essential for improving data quality in online research.
• Implementing effective blocking strategies can significantly enhance the reliability of behavioral science findings.