Flexibility in reaction time analysis: many roads to a false positive?

Area of Science:

• Cognitive Psychology
• Psychometrics
• Data Analysis

Background:

• Reaction times (RTs) are crucial measures in cognitive and psychological research.
• RT data often exhibit skewed distributions and potential outliers, necessitating specific analytical approaches.
• The flexibility in handling RT data is often overlooked in standard research practices.

Purpose of the Study:

• To investigate the impact of undisclosed analytical flexibility on the false-positive rate in reaction time studies.
• To quantify how choices in data preprocessing influence statistical outcomes.
• To raise awareness about the cumulative effect of analytical degrees of freedom.

Main Methods:

• Simulations were conducted using both real and synthetic reaction time datasets.
• Various preprocessing methods for handling RT data characteristics (e.g., distribution, outliers) were systematically combined.
• The resulting false-positive rates were analyzed across different combinations of preprocessing steps.

Main Results:

• The combination of multiple preprocessing methods significantly increased the false-positive rate.
• A false-positive rate as high as 17% was observed when several preprocessing techniques were applied.
• This indicates that flexibility in preprocessing steps can lead to an inflated Type I error rate.

Conclusions:

• Undisclosed flexibility in reaction time data preprocessing can substantially elevate false-positive rates.
• Researchers must be transparent about their preprocessing choices to ensure the validity of their findings.
• Further research is needed to explore the full extent of analytical flexibility and its impact on scientific conclusions.