Related Experiment Video

Updated: Aug 22, 2025

Combining Multiple Data Acquisition Systems to Study Corticospinal Output and Multi-segment Biomechanics
08:48

Combining Multiple Data Acquisition Systems to Study Corticospinal Output and Multi-segment Biomechanics

Published on: January 9, 2016

7.0K

Too Good to Be True: Bots and Bad Data From Mechanical Turk

Margaret A Webb1,2, June P Tangney2

  • 1Department of Criminology, Max-Planck Institute for the Study of Crime, Security, and Law Freiburg im Breisgau.

Perspectives on Psychological Science : a Journal of the Association for Psychological Science
|November 7, 2022
PubMed

Insights

Researchers caution against using Amazon

Area of Science:

  • Behavioral and social sciences research
  • Online data collection methodologies

Background:

  • Psychology increasingly relies on digital data sources.
  • Amazon's Mechanical Turk (MTurk) is a prominent platform for online studies.
  • In 2015, an estimated 45% of top behavioral and social science journals featured MTurk studies.

Purpose of the Study:

  • To report on the validity of data collected via Amazon's Mechanical Turk (MTurk).
  • To serve as a cautionary example for researchers utilizing MTurk for data collection.

Main Methods:

  • An online study was conducted using Amazon's Mechanical Turk (MTurk).
  • Data validity was assessed to determine the proportion of reliable participant responses.
  • Participant sample size was N = 529.

Main Results:

  • An estimated 2.6% of the participant sample (14 out of 529 individuals) provided valid data.
  • The overall data validity was significantly lower than anticipated.

Conclusions:

  • The findings suggest that data collected through MTurk may not always be reliable.
  • Researchers are urged to exercise caution and implement rigorous validation checks when using MTurk for data collection.
  • The platform's efficiency in data gathering may be offset by concerns regarding data quality.
Keywords:
Mechanical Turkcrowdsourcingdata qualityresearch data

More Related Videos

Shotgun Proteomics Sample Processing Automated by an Open-Source Lab Robot10:12

Shotgun Proteomics Sample Processing Automated by an Open-Source Lab Robot

Published on: October 28, 2021

3.8K
Automated, Quantitative Cognitive/Behavioral Screening of Mice: For Genetics, Pharmacology, Animal Cognition and Undergraduate Instruction16:23

Automated, Quantitative Cognitive/Behavioral Screening of Mice: For Genetics, Pharmacology, Animal Cognition and Undergraduate Instruction

Published on: February 26, 2014

14.4K

Related Concept Videos

Regression Toward the Mean01:52

Regression Toward the Mean

Regression toward the mean (“RTM”) is a phenomenon in which extremely high or low values—for example, and individual’s blood pressure at a particular moment—appear closer to a group’s average upon remeasuring. Although this statistical peculiarity is the result of random error and chance, it has been problematic across various medical, scientific, financial and psychological applications. In particular, RTM, if not taken into account, can interfere when...
6.3K
Mechanical Efficiency of Real Machines01:14

Mechanical Efficiency of Real Machines

The mechanical efficiency of a machine is a fundamental concept that describes how effectively a machine can convert input work into output work. According to this concept, the efficiency of a machine is equal to the ratio of the output work to the input work. An ideal machine, meaning a machine that has no energy losses, has an efficiency of one. This implies that the input work and the output work are equal.
However, in reality, no machine can be truly ideal, and all of them experience some...
808
Stereotype Content Model02:16

Stereotype Content Model

The Stereotype Content Model (SCM) was first proposed by Susan Fiske and her colleagues (Fiske, Cuddy, Glick & Xu, 2002; see also Fiske, 2012 and Fiske, 2017). The SCM specifies that when someone encounters a new group, they will stereotype them based on two metrics: warmth—or that group’s perceived intent, and how likely they are to provide help or inflict harm—and competence—or their ability to carry out that objective. Depending on the warmth-competence...
14.8K