When to Adjust for Multiple Testing: A Unifying Guiding Principle

Area of Science:

• Biostatistics
• Medical Research Methodology

Background:

• Medical literature frequently reports multiple statistical tests.
• Current guidance on adjusting for multiple testing is often unclear or contradictory.
• This ambiguity can hinder analysis, encourage poor practices, and reduce research credibility.

Purpose of the Study:

• To present a unifying principle for deciding when to adjust for multiple testing.
• To clarify the application of multiple testing adjustments in complex scenarios.

Main Methods:

• The study refines and illustrates a guiding principle for multiple testing adjustments.
• The principle focuses on the emphasis placed on results with small p-values.
• The approach is demonstrated in three complex multiple testing settings.

Main Results:

• A clear principle is proposed: adjust for multiple testing if and only if results are emphasized due to small p-values.
• This principle provides a framework for statisticians and researchers.
• The principle aids in selecting appropriate adjustment strategies.

Conclusions:

• The proposed principle offers clarity on multiple testing adjustments.
• Consistent application can improve the rigor and interpretability of medical research.
• This guidance aims to safeguard the credibility of published medical findings.