Zen and the art of multiple comparisons

Area of Science:

• Statistical inference
• Data-intensive research

Background:

• The challenge of multiple comparisons correction is increasingly relevant in modern data-intensive fields such as neuroimaging and genetics.
• Performing thousands of statistical tests without correction can lead to a high rate of false positives, compromising research reproducibility.

Purpose of the Study:

• To provide an introductory overview of hypothesis testing and multiple comparisons correction.
• To highlight the application of these methods in functional magnetic resonance imaging (fMRI) data analysis.
• To discuss the potential pitfalls of ignoring the multiple comparisons problem.

Main Methods:

• The article reviews fundamental concepts of hypothesis testing.
• It presents various principled techniques for correcting multiple statistical tests.
• Illustrative examples demonstrate the consequences of improper handling of multiple comparisons.

Main Results:

• Discussion of hypothesis testing and correction techniques for multiple comparisons.
• Demonstration of potential problems arising from unaddressed multiplicity.
• Exploration of effect size estimation in relation to multiple comparisons.

Conclusions:

• Failure to adequately address multiple comparisons significantly increases the risk of false positives.
• Erroneous conclusions are a direct consequence of neglecting proper statistical correction.
• Emphasizes the importance of robust statistical practices for research integrity.