Permutation P-values should never be zero: calculating exact P-values when permutations are randomly drawn.

Area of Science:

• Genomic research
• Statistical analysis
• Computational biology

Background:

• Permutation tests are widely used in genomics for statistical inference.
• Current methods for calculating p-values using permutation or Monte Carlo simulation are often flawed.
• This leads to understated p-values, particularly problematic in multiple testing scenarios.

Purpose of the Study:

• To identify and explain the common errors in calculating p-values from permutation tests in genomics.
• To propose a correct computational strategy for generating exact p-values from random permutations.
• To offer practical recommendations for implementing accurate permutation tests in genomic studies.

Main Methods:

• Review of existing literature on permutation testing and p-value calculation.
• Development of a novel computation strategy for exact p-values using random permutations.
• Validation of the strategy for diverse sample sizes and permutation numbers.

Main Results:

• Identified a systematic understatement of p-values by approximately 1/m (where m is the number of permutations) in common genomic applications.
• Demonstrated that permutation should be used to generate an exact discrete null distribution, not to estimate tail probabilities.
• Developed a universally applicable computation strategy for exact p-values.

Conclusions:

• Correcting the method of p-value calculation in permutation tests is crucial for accurate genomic research.
• The proposed strategy ensures exact p-values, mitigating the issue of understatement.
• Adoption of these methods will improve the reliability of statistical findings in genomics.