Big Data, Small Sample.

Area of Science:

• Statistics
• Genomics
• Bioinformatics

Background:

• Big Data studies, especially in genomics, face challenges with multiple comparisons and small sample sizes, impacting inference reliability.
• Current multiple testing procedures require very small tail probabilities, a condition rarely met in practice.
• Existing methods like permutation tests may not adequately control error rates in these scenarios.

Purpose of the Study:

• To investigate the impact of non-standard sampling distributions on error rates in multiple testing.
• To assess the reliability of statistical inference in genomic studies with large numbers of tests and limited sample sizes.
• To explore advanced statistical methods for improving error rate control.

Main Methods:

• Utilizing Edgeworth expansions to approximate sampling distributions beyond typical assumptions.
• Analyzing departures from standard distributional assumptions and their effect on actual error rates.
• Reviewing commonly used methods like permutation tests and finite sampling inequalities.

Main Results:

• Actual error rates can significantly deviate from nominal levels, leading to excessive false positives.
• The condition for error rate control based on large deviation theory is frequently not satisfied.
• Edgeworth expansions demonstrate potential for higher-order approximations to sampling distributions.

Conclusions:

• Widespread problems with error rate control, particularly inflated false positive rates, may exist in Big Data and genomic studies.
• This lack of reliability contributes to the "reproducibility crisis" in science.
• Edgeworth expansions offer a promising approach to enhance the reliability of studies with numerous comparisons and modest sample sizes.