Remote data collection and cognitive task performance in the age of internet bad actors

Bad actors and bots present threats to data integrity in remote research designs, compromising estimates of cognitive task performance. Here, we share our experiences and recommendations for identifying bad actors and bots and preventing them from enrolling in remote studies of cognitive task performance. As part of a separate large-scale validation study, 3488 participants were recruited through various platforms, including social media and MTurk, and were compensated for their participation. When inspecting these data, we observed a high incidence of unusual or nonsensical responses to demographics questions in combination with unusual cognitive performance patterns. In response, participants were categorized according to bad actor status as determined by 1) number and risk-level associated with red flags within initial screening and pattern of responses; and 2) a researcher review process to identify patterns of aberrant, patterned, or inconsistent responding. Participants were either automatically labeled as bad actors (n = 2829), labeled as bad actors after researcher review (n = 242), or labeled as good actors after researcher review (n = 417). Cognitive task performance was assessed with Adaptive Cognitive Evaluation-Explorer (ACE-X), a mobile, gamified collection of executive function tasks including measures of working memory, attention, and cognitive flexibility. We considered differences between good and bad actors for mean response times, variability in response times, accuracy, and number of trials attempted. Across 6 of the 7 ACE-X tasks evaluated, bad actors responded slower than good actors on average (β = 34.10-150.15, p < 0.001-0.047). Additional differences in task performance suggested that overall, bad actors also tended to be more variable in their response times (β = -0.32-1.01, p < 0.001), less accurate (β = -2.15-0.00, p < 0.001- > 0.999), and responded to fewer trials (β = -1.30-0.00, p < 0.001- > 0.999). Findings presented here suggest that bad actors show consistent differences in their patterns of performance on cognitive assessments, and therefore careful consideration related to study design and protocols to ensure data integrity are needed when evaluating cognitive performance in remote research settings.