Why you shouldn't trust data collected on MTurk

Several prior studies have used advanced methodological techniques to demonstrate that there is an issue with the quality of data that can be collected on Amazon's Mechanical Turk (MTurk). The goal of the present project was to provide an accessible demonstration of this issue. We administered 27 semantic antonyms-pairs of items that assess clearly contradictory content (e.g., "I talk a lot" and "I rarely talk")-to samples drawn from Connect (N₁ = 100), Prolific (N₂ = 100), and MTurk (N₃ = 400; N₄ = 600). Despite most of these item pairs being negatively correlated on Connect and Prolific, over 96% were positively correlated on MTurk. This issue could not be remedied by screening the data using common attention check measures nor by recruiting only "high-productivity" and "high-reputation" participants. These findings provide clear evidence that data collected on MTurk simply cannot be trusted.