Jove
Visualize
Contact Us
JoVE
x logofacebook logolinkedin logoyoutube logo
ABOUT JoVE
OverviewLeadershipBlogJoVE Help Center
AUTHORS
Publishing ProcessEditorial BoardScope & PoliciesPeer ReviewFAQSubmit
LIBRARIANS
TestimonialsSubscriptionsAccessResourcesLibrary Advisory BoardFAQ
RESEARCH
JoVE JournalMethods CollectionsJoVE Encyclopedia of ExperimentsArchive
EDUCATION
JoVE CoreJoVE BusinessJoVE Science EducationJoVE Lab ManualFaculty Resource CenterFaculty Site
Terms & Conditions of Use
Privacy Policy
Policies

Related Concept Videos

Reliability and Validity01:29

Reliability and Validity

13.7K
Reliability and validity are two important considerations that must be made with any type of data collection. Reliability refers to the ability to consistently produce a given result. In the context of psychological research, this would mean that any instruments or tools used to collect data do so in consistent, reproducible ways.
13.7K
Surveys02:16

Surveys

16.6K
Often, psychologists develop surveys as a means of gathering data. Surveys are lists of questions to be answered by research participants, and can be delivered as paper-and-pencil questionnaires, administered electronically, or conducted verbally. Generally, the survey itself can be completed in a short time, and the ease of administering a survey makes it easy to collect data from a large number of people.
16.6K
Naturalistic Observations02:30

Naturalistic Observations

17.0K
If you want to understand how behavior occurs, one of the best ways to gain information is to simply observe the behavior in its natural context. However, people might change their behavior in unexpected ways if they know they are being observed. How do researchers obtain accurate information when people tend to hide their natural behavior? As an example, imagine that your professor asks everyone in your class to raise their hand if they always wash their hands after using the restroom. Chances...
17.0K
Systematic Error: Methodological and Sampling Errors01:15

Systematic Error: Methodological and Sampling Errors

8.6K
In the case of systematic errors, the sources can be identified, and the errors can be subsequently minimized by addressing these sources. According to the source, systematic errors can be divided into sampling, instrumental, methodological, and personal errors.
Sampling errors originate from improper sampling methods or the wrong sample population. These errors can be minimized by refining the sampling strategy. Defective instruments or faulty calibrations are the sources of instrumental...
8.6K
Longitudinal Research02:20

Longitudinal Research

13.0K
Sometimes we want to see how people change over time, as in studies of human development and lifespan. When we test the same group of individuals repeatedly over an extended period of time, we are conducting longitudinal research. Longitudinal research is a research design in which data-gathering is administered repeatedly over an extended period of time. For example, we may survey a group of individuals about their dietary habits at age 20, retest them a decade later at age 30, and then again...
13.0K
Regression Toward the Mean01:52

Regression Toward the Mean

6.8K
Regression toward the mean (“RTM”) is a phenomenon in which extremely high or low values—for example, and individual’s blood pressure at a particular moment—appear closer to a group’s average upon remeasuring. Although this statistical peculiarity is the result of random error and chance, it has been problematic across various medical, scientific, financial and psychological applications. In particular, RTM, if not taken into account, can interfere when...
6.8K

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by
Same author

Assessing Conspiracist Ideation Reliably, Validly, and Efficiently: A Psychometric Comparison of Five Short-Form Measures.

Assessment·2025
Same author

Validating the IDRIS and IDRIA: Two infrequency/frequency scales for detecting careless and insufficient effort survey responders.

Behavior research methods·2024
Same author

Moral Foundations Partially Explain the Associations of Machiavellianism, Grandiose Narcissism, and Psychopathy With Homonegativity and Transnegativity.

Journal of homosexuality·2022
Same author

Antecedents and consequences of COVID-19 conspiracy beliefs: A systematic review.

Social science & medicine (1982)·2022
Same author

Animal House: The Dark Tetrad traits and membership in sororities and fraternities.

Acta psychologica·2021
Same journal

Relations of social cognition with affective states: Insights from an expanded 2650-word database on warmth and competence.

Behavior research methods·2026
Same journal

The evaluation of devaluation: Deficient outcome devaluation leads to wrongly considering goal-directed actions as habits.

Behavior research methods·2026
Same journal

Author Correction: TCBLex - A lexical database of Finnish literary texts for children.

Behavior research methods·2026
Same journal

Subjective norms of paintings: Integrating perceptual, cognitive, and emotional dimensions.

Behavior research methods·2026
Same journal

The Rhythm Reproduction Task for children: A psychometric examination.

Behavior research methods·2026
Same journal

QualGames: A Qualtrics implementation and a database of behavioral game theory tasks.

Behavior research methods·2026
See all related articles

Related Experiment Video

Updated: Jan 11, 2026

A Modified Trier Social Stress Test for Vulnerable Mexican American Adolescents
06:15

A Modified Trier Social Stress Test for Vulnerable Mexican American Adolescents

Published on: July 10, 2017

13.4K

Why you shouldn't trust data collected on MTurk.

Cameron S Kay1,2

  • 1Environmental Social Sciences Department, Stanford University, 473 Via Ortega, Stanford, CA, 94301, USA. cameronstuartkay@gmail.com.

Behavior Research Methods
|November 10, 2025
PubMed
Summary
This summary is machine-generated.

Data quality on Amazon Mechanical Turk (MTurk) is unreliable. Contradictory survey items showed positive correlations on MTurk, unlike other platforms, indicating untrustworthy data collection.

Keywords:
Amazon’s Mechanical TurkCareless respondingData qualityInvalid respondingPsychometricsSurvey design and methodology

More Related Videos

Methods to Test Visual Attention Online
09:44

Methods to Test Visual Attention Online

Published on: February 19, 2015

12.4K
Revised and Neuroimaging-Compatible Versions of the Dual Task Screen
07:52

Revised and Neuroimaging-Compatible Versions of the Dual Task Screen

Published on: October 5, 2020

3.9K

Related Experiment Videos

Last Updated: Jan 11, 2026

A Modified Trier Social Stress Test for Vulnerable Mexican American Adolescents
06:15

A Modified Trier Social Stress Test for Vulnerable Mexican American Adolescents

Published on: July 10, 2017

13.4K
Methods to Test Visual Attention Online
09:44

Methods to Test Visual Attention Online

Published on: February 19, 2015

12.4K
Revised and Neuroimaging-Compatible Versions of the Dual Task Screen
07:52

Revised and Neuroimaging-Compatible Versions of the Dual Task Screen

Published on: October 5, 2020

3.9K

Area of Science:

  • Psychological science
  • Data science
  • Human-computer interaction

Background:

  • Prior research indicates data quality issues on Amazon Mechanical Turk (MTurk).
  • Advanced methods have confirmed these data integrity problems.
  • Accessible demonstrations are needed to illustrate the extent of the issue.

Purpose of the Study:

  • To provide a clear and accessible demonstration of data quality problems on MTurk.
  • To compare data quality across MTurk, Connect, and Prolific platforms.
  • To investigate the reliability of semantic antonym item pairs for assessing data integrity.

Main Methods:

  • Administered 27 semantic antonym item pairs assessing contradictory content.
  • Collected data from samples on Connect (N=100), Prolific (N=100), and MTurk (N=1000).
  • Analyzed item pair correlations across different platforms.

Main Results:

  • Over 96% of item pairs were positively correlated on MTurk, contrasting with negative correlations on Connect and Prolific.
  • Data screening using attention checks did not resolve the issue.
  • Recruiting high-productivity or high-reputation MTurk participants did not improve data reliability.

Conclusions:

  • Data collected via MTurk exhibits significant quality and reliability issues.
  • Standard data cleaning and participant selection methods are insufficient to ensure MTurk data integrity.
  • Findings strongly suggest that MTurk data cannot be trusted for research purposes.