Concerns and recommendations for using Amazon MTurk for eating disorder research.

Area of Science:

• Psychiatry and Psychology
• Public Health
• Sociology

Background:

• Online recruitment platforms like Amazon Mechanical Turk (MTurk) are increasingly used for research, including studies on eating disorders (EDs).
• Validating common ED symptom measures in diverse populations, such as transgender adults, is crucial for accurate clinical assessment and intervention.
• Concerns exist regarding the data quality and reliability of participant responses obtained through crowdsourcing platforms.

Purpose of the Study:

• To validate and norm common eating disorder symptom measures in a large, representative community sample of transgender adults in the United States.
• To evaluate the effectiveness of Amazon Mechanical Turk (MTurk) as a recruitment platform for this population and assess the quality of collected data.
• To identify potential issues with data validity and participant representation in online research.

Main Methods:

• Recruited 2,250 transgender adults across the United States via MTurk.
• Administered demographic questionnaires, the Eating Disorder Examination-Questionnaire, and the Eating Attitudes Test-26.
• Implemented a comprehensive set of attention and validity checks, alongside analysis of response concordance and qualitative data.

Main Results:

• Standard validity and attention checks were insufficient to detect most invalid responses.
• Analysis of multiple ED measures, weight history, and gender identity revealed significant data discrepancies.
• Qualitative data suggested economic pressures on MTurk workers may compel misrepresentation, compromising data integrity.

Conclusions:

• The majority of data collected via MTurk for this study were likely invalid, raising concerns about previous research using this platform.
• Researchers must exercise extreme caution and implement rigorous methods to mitigate and identify invalid data when using MTurk for eating disorder research.
• Suggestions are provided for improving data quality and reliability in online research settings.