Do mturkers collude in interactive online experiments?

Area of Science:

• Behavioral Economics
• Experimental Psychology
• Human-Computer Interaction

Background:

• Internal validity of online experiments can be threatened by participant collusion.
• Crowdsourcing platforms (e.g., Amazon Mechanical Turk) are widely used for participant recruitment.
• Understanding collusion prevalence is crucial for ensuring data quality in online research.

Purpose of the Study:

• To measure the prevalence of collusion among MTurk workers in interactive online experiments.
• To investigate whether collusion is influenced by specific experimental design choices.
• To assess the risk of collusion to data validity in crowdsourced online experiments.

Main Methods:

• Conducted two experiments recruiting participants via the MTurk crowdsourcing platform.
• Manipulated experimental design features to test their impact on collusion.
• Analyzed participant behavior to detect evidence of collusion.

Main Results:

• No evidence of collusion was found in treatments mimicking typical interactive online experiment designs.
• Approximately 3% of MTurk workers engaged in collusion when offered unusually high payoffs for doing so.
• Collusion is not a widespread issue in standard online experiments but can occur under specific high-incentive conditions.

Conclusions:

• Typical interactive online experiments using MTurk workers are likely not significantly affected by collusion.
• Collusion presents a potential threat to data validity in online experiments when strong incentives are present.
• Researchers should remain vigilant about collusion risks, particularly in high-stakes online experimental designs.