Related Experiment Video
Updated: Mar 18, 2026
Evaluating Usability Aspects of a Mixed Reality Solution for Immersive Analytics in Industry 4.0 Scenarios
Published on: October 6, 2020
Measuring the Prevalence of Problematic Respondent Behaviors among MTurk, Campus, and Community Participants
Elizabeth A Necka1, Stephanie Cacioppo2, Greg J Norman1
1University of Chicago, Chicago, Illinois, United States of America.
Abstract:
The reliance on small samples and underpowered studies may undermine the replicability of scientific findings. Large sample sizes may be necessary to achieve adequate statistical power. Crowdsourcing sites such as Amazon's Mechanical Turk (MTurk) have been regarded as an economical means for achieving larger samples. Because MTurk participants may engage in behaviors which adversely affect data quality, much recent research has focused on assessing the quality of data obtained from MTurk samples. However, participants from traditional campus- and community-based samples may also engage in behaviors which adversely affect the quality of the data that they provide. We compare an MTurk, campus, and community sample to measure how frequently participants report engaging in problematic respondent behaviors. We report evidence that suggests that participants from all samples engage in problematic respondent behaviors with comparable rates. Because statistical power is influenced by factors beyond sample size, including data integrity, methodological controls must be refined to better identify and diminish the frequency of participant engagement in problematic respondent behaviors.
Insights
Problematic respondent behaviors occur at similar rates across Amazon Mechanical Turk (MTurk), campus, and community samples. Enhancing methodological controls is crucial for data integrity and statistical power, regardless of sample source.
Area of Science:
- Psychology
- Research Methodology
- Data Science
Background:
- Replicability of scientific findings is threatened by small, underpowered studies.
- Crowdsourcing platforms like Amazon Mechanical Turk (MTurk) offer economical large sample sizes.
- Concerns exist regarding data quality from MTurk participants due to problematic behaviors.
Purpose of the Study:
- To compare the frequency of problematic respondent behaviors across MTurk, campus, and community samples.
- To assess if MTurk samples exhibit uniquely poor data quality compared to traditional samples.
- To inform strategies for improving data integrity in diverse research settings.
Main Methods:
- A comparative study design was employed.
- Participants were recruited from three distinct sources: MTurk, university campus, and the general community.
- Data were collected on the self-reported engagement in problematic respondent behaviors.
Main Results:
- Problematic respondent behaviors were reported at comparable rates across all three sample types (MTurk, campus, community).
- No significant difference in the frequency of problematic behaviors was observed between the samples.
- This suggests that data quality concerns are not exclusive to crowdsourced samples.
Conclusions:
- Traditional sampling methods do not inherently guarantee higher data integrity than crowdsourcing.
- Statistical power is influenced by data integrity, not solely sample size.
- Methodological controls must be refined to detect and mitigate problematic respondent behaviors across all sample sources.
Related Concept Videos
02:16Surveys
01:52Regression Toward the Mean