Multiverse simulation to explore the impact of analytical choices on type I and type II errors in a reaction time...

Area of Science:

• Cognitive Psychology
• Social Sciences
• Data Analysis

Background:

• Researcher degrees of freedom in data analysis pose challenges, leading to varied conclusions in social sciences.
• The congruency sequence effect (CSE) is a well-studied phenomenon in cognitive control research.

Purpose of the Study:

• To propose and demonstrate an exploratory multiverse simulation to assess analytical decision pathway sensitivity.
• To empirically compare different analytical choices and their impact on research findings.

Main Methods:

• Reviewed literature for common, non-theory-specific analytical decisions (e.g., outlier exclusion, hypothesis testing).
• Developed a simulation framework incorporating these decisions.
• Generated 20,000 simulated datasets and performed 1 million analyses across 50 decision pathways.

Main Results:

• Substantial differences in statistical power and type I error rates were observed across analytical pathways.
• Certain analytical pathways presented a significant risk of high false positive rates.
• Results indicate sensitivity of the congruency sequence effect (CSE) to analytical choices.

Conclusions:

• Extensive simulations are crucial for investigating data handling and hypothesis testing in fields with high analytical variability.
• Transparency in analytical decisions is vital for robust and reproducible research.
• This multiverse simulation approach can serve as a model for other research areas facing similar challenges.