Significance levels for studies with correlated test statistics

Area of Science:

• Statistics
• Bioinformatics
• Genomics

Background:

• Assessing evidence against a global null hypothesis is crucial when testing numerous hypotheses.
• Permutation-based methods are commonly used to evaluate the statistical significance of the largest test statistic.
• Correlated test statistics can lead to misleading histograms and inaccurate tail counts, as noted by Efron (2007).

Purpose of the Study:

• To demonstrate that permutation-based estimates of overall significance levels can be misleading when test statistics are correlated.
• To propose a novel method for obtaining conditional significance levels by conditioning on histogram spread.
• To provide a statistically justified approach using the conditionality principle (Cox and Hinkley, 1974).

Main Methods:

• Developed a method to condition permutation-based significance level estimates on a measure of histogram spread.
• Applied the conditionality principle to justify the proposed statistical approach.
• Illustrated the method's application using gene expression data.

Main Results:

• Showed that correlated test statistics can indeed lead to misleading permutation-based significance level estimates.
• Introduced a practical method for calculating conditional significance levels.
• Demonstrated the necessity and utility of conditional significance levels in real-world data analysis.

Conclusions:

• Conditional significance levels are essential for accurate inference when dealing with correlated test statistics in large-scale hypothesis testing.
• The proposed method offers a robust alternative to standard permutation tests in such scenarios.
• This approach is particularly relevant for analyzing complex biological data, such as gene expression profiles.