Comparing multiple comparisons: practical guidance for choosing the best multiple comparisons test.

Area of Science:

• Statistics
• Ecology
• Biostatistics

Background:

• Multiple comparisons tests (MCTs) are crucial for analyzing group differences after significant findings in linear models.
• Decades of research have yielded numerous MCTs, yet their correct application and reporting remain challenging.
• MCTs are widely used across disciplines, with over 40,000 documented uses in ecological journals in the past 60 years.

Purpose of the Study:

• To evaluate 17 different multiple comparisons tests (MCTs) for their appropriate use and performance.
• To provide evidence-based recommendations for selecting MCTs based on data characteristics and research questions.
• To clarify best practices for planned versus unplanned comparisons and specific statistical test applications.

Main Methods:

• A comprehensive literature review was conducted to identify recommendations for the correct use of MCTs.
• A simulation study was performed to evaluate the performance of nine common MCTs with overlapping usage.
• Code and data for the simulation are publicly available to ensure transparency and reproducibility.

Main Results:

• Planned comparisons are strongly recommended over unplanned comparisons.
• Mann-Whitney-Wilcoxon U test is recommended for planned non-parametric comparisons.
• Scheffé's S test is advised for any linear combination of unplanned means; Tukey's HSD, Bonferroni, or Dunn-Sidak tests are recommended for pairwise comparisons.

Conclusions:

• The study provides clear guidelines for selecting appropriate multiple comparisons tests.
• Adherence to recommended MCTs can improve the accuracy and interpretability of statistical analyses.
• The findings aim to enhance the correct use and reporting of MCTs in scientific research.