Comparing Health Survey Data Cost and Quality Between Amazon's Mechanical Turk and Ipsos' KnowledgePanel:...

Area of Science:

• Health Services Research
• Survey Methodology
• Data Science

Background:

• Web-based survey data collection offers diverse options, including probability-based panels and less expensive convenience panels.
• Convenience panels, like Amazon's Mechanical Turk (MTurk), vary in respondent representativeness and motivation.
• Researchers must weigh data quality against cost savings when choosing between panel types.

Purpose of the Study:

• To compare the relative costs and data quality of fielding equivalent surveys on MTurk and KnowledgePanel.
• To assess the impact of quality assurance steps on MTurk data.
• To evaluate the representativeness and psychometric properties of data from both platforms.

Main Methods:

• Equivalent surveys were administered to MTurk and KnowledgePanel participants.
• Quality assurance steps were applied to MTurk data; standard protocols were used for KnowledgePanel.
• Data quality was assessed using metrics like consistency, reliability, representativeness, and correlations, including the use of fake conditions to identify honest respondents.

Main Results:

• MTurk data quality improved with prescreening but generally remained inferior to KnowledgePanel.
• Weighted MTurk demographic estimates matched national averages, while KnowledgePanel estimates were closer even before weighting.
• Correlations between Patient-Reported Outcomes Measurement Information System (PROMIS) measures and demographics differed significantly, particularly for educational level on MTurk.

Conclusions:

• MTurk data quality is often inferior to KnowledgePanel, but cost benefits may outweigh quality differences for certain research applications.
• Caution is advised when using MTurk for studies where educational level is a critical variable.
• The study highlights the trade-offs between cost and data quality in online survey research.