Inattentive responding can induce spurious associations between task behaviour and symptom measures

Area of Science:

• Psychiatric research methodology
• Cognitive psychology
• Online data collection

Background:

• Online samples offer advantages for psychiatric research but present unique challenges.
• Potential pitfalls, such as spurious correlations, are not fully understood.
• Asymmetric score distributions in psychiatric surveys can be influenced by careless responding.

Purpose of the Study:

• To detail circumstances leading to spurious correlations between task behavior and symptom scores in online psychiatric research.
• To investigate the impact of careless responding on these correlations.
• To evaluate methods for mitigating spurious findings.

Main Methods:

• Analysis of two online samples (N=779) performing common cognitive tasks.
• Examination of survey data for careless responding.
• Assessment of task performance data for inattention.
• Statistical analysis to identify correlations between symptom scores and task behavior.

Main Results:

• Spurious correlations were observed between symptom scores and task behavior in online samples.
• False-positive rates for these correlations increased with sample size.
• Excluding participants based on survey carelessness eliminated spurious correlations.
• Exclusion based solely on task performance was less effective in removing these false associations.

Conclusions:

• Careless responding in psychiatric surveys can create artificial associations with cognitive task performance in online studies.
• Researchers must be cautious of spurious correlations, particularly with large online datasets.
• Implementing checks for careless survey responding is crucial for data integrity in online psychiatric research.