Reputation as a sufficient condition for data quality on Amazon Mechanical Turk

Area of Science:

• Psychology
• Human-Computer Interaction
• Data Science

Background:

• Crowdsourcing platforms like Amazon Mechanical Turk (MTurk) are increasingly used for online behavioral studies.
• Ensuring data quality from crowdsourced participants is a significant challenge.

Purpose of the Study:

• To compare the effectiveness of attention check questions (ACQs) and high-reputation worker selection for improving data quality on MTurk.
• To determine the optimal strategy for obtaining reliable data in crowdsourced research.

Main Methods:

• Experiment 1: Compared data quality between high-reputation (>=95% approval) and low-reputation MTurk workers, with and without ACQs.
• Experiment 2: Further investigated the impact of worker reputation and productivity on data quality.

Main Results:

• High-reputation workers consistently provided higher-quality data and rarely failed ACQs.
• ACQs only marginally improved data quality for low-reputation workers.
• More productive, high-reputation workers yielded the best data quality.

Conclusions:

• Selecting MTurk workers with high reputation is a more effective method for ensuring high-quality data than using ACQs.
• Relying on ACQs may introduce selection bias by excluding participants, potentially skewing results.