Are the attention checks embedded in delay discounting tasks a valid marker for data quality?

Area of Science:

• Psychology
• Behavioral Economics
• Research Methodology

Background:

• Crowdsourcing platforms are increasingly used for research, necessitating methods to ensure data quality.
• Attention checks are commonly embedded in delay discounting (DD) tasks to filter participants.
• The validity of DD-specific attention checks as a general measure of data quality remains underexplored.

Purpose of the Study:

• To evaluate the validity of attention checks used in delay discounting (DD) tasks.
• To determine if DD attention check performance predicts other measures of data quality.
• To assess if failing DD attention checks is associated with participant characteristics or discounting behavior.

Main Methods:

• Utilized data from two studies with a total of 700 participants (N = 700).
• Assessed DD attention check performance against non-DD attention checks and various data quality indicators.
• Examined associations between failing DD attention checks and discounting rates or participant characteristics.

Main Results:

• Failing DD attention checks correlated with a higher likelihood of nonsystematic DD data.
• DD attention checks demonstrated inadequate discriminability and failure sometimes linked to individual differences, suggesting potential bias from exclusion.
• Performance on DD attention checks did not correlate with performance on other attention checks or general data quality indicators.

Conclusions:

• DD attention checks alone are insufficient indicators of overall data quality for DD tasks or general survey data.
• Current DD attention checks may inaccurately exclude participants and introduce bias.
• Future research should focus on developing and validating more robust attention checks and data cleaning strategies for crowdsourced research.