Influence of multiple hypothesis testing on reproducibility in neuroimaging research: A simulation study and...

Area of Science:

• Neuroscience
• Psychology
• Biostatistics

Background:

• Research reproducibility is a growing concern in psychology and neurosciences.
• Simultaneous testing of multiple hypotheses can lead to false positives without proper p-value correction.
• The multiple testing problem remains a significant theoretical and practical challenge.

Purpose of the Study:

• To assess the reproducibility of simulated experiments under multiple testing conditions.
• To compare the performance of various statistical methods for controlling error rates.
• To identify strategies for improving the reproducibility of research findings.

Main Methods:

• Simulated experiments were used to evaluate methods controlling family-wise error rate (FWER) and false discovery rate (FDR).
• Techniques included random field theory (RFT), cluster-mass permutation testing, adaptive FDR, and classical methods.
• Two different simulation models were employed to assess method performance.

Main Results:

• Permutation testing demonstrated the highest statistical power among the evaluated multiple testing methods.
• Grouping hypotheses based on prior knowledge was found to increase statistical power.
• Equal emphasis on primary and follow-up studies yielded the most reproducible outcomes.

Conclusions:

• Strict multiple testing corrections alone are insufficient to guarantee reproducibility in neuroimaging.
• The study introduces "MultiPy," an open-source Python toolkit for multiple hypothesis testing.
• The findings aim to enhance statistical data analysis practices and aid in power/reproducibility analyses for future experiments.