Evaluating Amazon's Mechanical Turk as a tool for experimental behavioral research

Area of Science:

• Cognitive Psychology
• Experimental Psychology
• Human-Computer Interaction

Background:

• Amazon Mechanical Turk (AMT) is a crowdsourcing platform increasingly used for psychological data collection.
• Traditional laboratory studies offer controlled environments, whereas AMT presents challenges in controlling the testing environment.
• Cognitive behavioral experiments require sustained attention, complex instructions, and precise timing, posing unique demands for online platforms.

Purpose of the Study:

• To empirically evaluate the fidelity of Amazon Mechanical Turk for cognitive behavioral experiments.
• To assess the suitability of AMT for tasks requiring multiple trials and millisecond accuracy.
• To identify challenges and provide insights for researchers using AMT in psychological studies.

Main Methods:

• Replication of diverse experimental psychology tasks (e.g., Stroop, Flanker, Posner Cuing) on AMT.
• Recruitment of participants through the Amazon Mechanical Turk platform.
• Comparison of results from online AMT replications with traditional laboratory findings.

Main Results:

• Most task replications were qualitatively successful, validating the use of AMT for online data collection.
• Disparities were observed between laboratory results and online results for certain cognitive tasks.
• The study identified key lessons and considerations for conducting psychological experiments on AMT.

Conclusions:

• Amazon Mechanical Turk is a viable, albeit imperfect, tool for cognitive behavioral research.
• Researchers must be aware of potential limitations and discrepancies when using AMT compared to lab settings.
• The findings offer practical guidance for optimizing the use of crowdsourcing in psychological research.