Multiple comparisons: To compare or not to compare, that is the question

Area of Science:

• Statistics
• Experimental Design

Background:

• Minimizing Type-I errors is crucial in research.
• Single statistical tests allow direct control of Type-I error rates.
• Multiple comparisons significantly inflate the risk of Type-I errors.

Purpose of the Study:

• To review the advantages and disadvantages of adjusting p-value thresholds for multiple comparisons.
• To discuss familiar-wise and experiment-wise error rates.
• To provide guidance on when to apply or avoid p-value adjustments.

Main Methods:

• Literature review and commentary on statistical adjustment methods.
• Discussion of common adjustments, such as the Bonferroni adjustment.
• Analysis of the risks associated with rigid adjustments.

Main Results:

• While adjustments like Bonferroni aim to control Type-I errors, they can be rigid and arbitrarily applied.
• Multiple comparisons necessitate careful consideration of error rates.
• The effectiveness and appropriateness of adjustments vary.

Conclusions:

• Adjustments for multiple comparisons have benefits but also drawbacks.
• Researchers should judiciously decide when to implement p-value threshold adjustments.
• Understanding error rates is key to appropriate statistical practice.