The potential for increased power from combining P-values testing the same hypothesis

Area of Science:

• Statistics
• Biostatistics
• Hypothesis Testing

Background:

• Traditional hypothesis testing uses a single, prespecified test statistic.
• Identifying the single most powerful test statistic is often challenging.
• Multiple relevant test statistics may exist for evaluating treatment effects.

Purpose of the Study:

• To introduce and evaluate a method for combining p-values from multiple test statistics for enhanced statistical inference.
• To demonstrate the increased power of combined p-value approaches compared to single test statistics.
• To explore the applicability of this method in scenarios with more covariates than observations.

Main Methods:

• Utilizing randomization-based tests to combine p-values from multiple prespecified test statistics.
• Comparing the power of combined p-value methods (e.g., Fisher's combination, minimum p-value) against single test statistics and Simes's method.
• Investigating the method's performance when the number of covariates exceeds the number of observations.

Main Results:

• Combining p-values from multiple test statistics significantly increases statistical power compared to using a single test.
• The proposed method demonstrates remarkable power gains.
• The approach is versatile and applicable even when the number of covariates is greater than the number of observations.

Conclusions:

• Combining p-values from multiple test statistics provides a more powerful and flexible approach to hypothesis testing.
• This method is preferable to relying on a single p-value due to substantial power increases.
• Limitations include the lack of an unbiased treatment effect estimator and inapplicability to models with treatment-by-covariate interactions.