Related Experiment Video

Updated: May 4, 2026

Evaluating Usability Aspects of a Mixed Reality Solution for Immersive Analytics in Industry 4.0 Scenarios
06:02

Evaluating Usability Aspects of a Mixed Reality Solution for Immersive Analytics in Industry 4.0 Scenarios

Published on: October 6, 2020

2.1K

Reputation as a sufficient condition for data quality on Amazon Mechanical Turk

Eyal Peer1, Joachim Vosgerau, Alessandro Acquisti

  • 1Graduate School of Business Administration, Bar-Ilan University, Ramat-Gan, Israel, 52900, eyal.peer@biu.ac.il.

Behavior Research Methods
|December 21, 2013
PubMed

Insights

Ensuring high-quality data in crowdsourced studies is crucial. Sampling high-reputation workers on Amazon Mechanical Turk (MTurk) effectively improves data quality without needing attention check questions (ACQs).

Area of Science:

  • Psychology
  • Human-Computer Interaction
  • Data Science

Background:

  • Crowdsourcing platforms like Amazon Mechanical Turk (MTurk) are increasingly used for online behavioral studies.
  • Ensuring data quality from crowdsourced participants is a significant challenge.

Purpose of the Study:

  • To compare the effectiveness of attention check questions (ACQs) and high-reputation worker selection for improving data quality on MTurk.
  • To determine the optimal strategy for obtaining reliable data in crowdsourced research.

Main Methods:

  • Experiment 1: Compared data quality between high-reputation (>=95% approval) and low-reputation MTurk workers, with and without ACQs.
  • Experiment 2: Further investigated the impact of worker reputation and productivity on data quality.

Main Results:

  • High-reputation workers consistently provided higher-quality data and rarely failed ACQs.
  • ACQs only marginally improved data quality for low-reputation workers.
  • More productive, high-reputation workers yielded the best data quality.

Conclusions:

  • Selecting MTurk workers with high reputation is a more effective method for ensuring high-quality data than using ACQs.
  • Relying on ACQs may introduce selection bias by excluding participants, potentially skewing results.

More Related Videos

Humor or Rationality? The Neural Mechanisms of How Agent Type and Language Style Influence Satisfaction with Ride-Hailing Service Failure Recovery09:53

Humor or Rationality? The Neural Mechanisms of How Agent Type and Language Style Influence Satisfaction with Ride-Hailing Service Failure Recovery

Published on: March 13, 2026

104
Evaluating Tests of Cognition using a Computerized Touch-Sensitive Tablet, Eye Tracking, and Functional Magnetic Resonance Imaging10:10

Evaluating Tests of Cognition using a Computerized Touch-Sensitive Tablet, Eye Tracking, and Functional Magnetic Resonance Imaging

Published on: January 30, 2026

660

Related Concept Videos

Regression Toward the Mean01:52

Regression Toward the Mean

Regression toward the mean (“RTM”) is a phenomenon in which extremely high or low values—for example, and individual’s blood pressure at a particular moment—appear closer to a group’s average upon remeasuring. Although this statistical peculiarity is the result of random error and chance, it has been problematic across various medical, scientific, financial and psychological applications. In particular, RTM, if not taken into account, can interfere when...
6.3K
Stereotype Content Model02:16

Stereotype Content Model

The Stereotype Content Model (SCM) was first proposed by Susan Fiske and her colleagues (Fiske, Cuddy, Glick & Xu, 2002; see also Fiske, 2012 and Fiske, 2017). The SCM specifies that when someone encounters a new group, they will stereotype them based on two metrics: warmth—or that group’s perceived intent, and how likely they are to provide help or inflict harm—and competence—or their ability to carry out that objective. Depending on the warmth-competence...
13.1K
Reliability and Validity01:29

Reliability and Validity

Reliability and validity are two important considerations that must be made with any type of data collection. Reliability refers to the ability to consistently produce a given result. In the context of psychological research, this would mean that any instruments or tools used to collect data do so in consistent, reproducible ways.
12.9K
Quantifying and Rejecting Outliers: The Grubbs Test01:02

Quantifying and Rejecting Outliers: The Grubbs Test

Sometimes, a data set can have a recorded numerical observation that greatly  deviates from the rest of the data. Assuming that the data is normally distributed, a statistical method called the Grubbs test can be used to determine whether the observation is truly an outlier.  To perform a two-tailed Grubbs test, first, calculate the absolute difference between the outlier and the mean. Then, calculate the ratio between this difference and the standard deviation of the sample. This...
4.0K
Detection of Gross Error: The Q Test01:00

Detection of Gross Error: The Q Test

When one or more data points appear far from the rest of the data, there is a need to determine whether they are outliers and whether they should be eliminated from the data set to ensure an accurate representation of the measured value. In many cases, outliers arise from gross errors (or human errors) and do not accurately reflect the underlying phenomenon. In some cases, however, these apparent outliers reflect true phenomenological differences. In these cases, we can use statistical methods...
7.1K
Kendall's Tau Test01:16

Kendall's Tau Test

Kendall's tau test, also known as the Kendall rank coefficient test, is a nonparametric method for assessing association between two variables. This test is particularly useful for identifying significant correlations when the distributions of the sample and population are unknown. Developed in 1938 by the British statistician Sir Maurice George Kendall, the tau coefficient (denoted as τ) serves as a rank correlation coefficient, with values ranging from -1 to +1.
A τ value of +1...
1.1K