How well do multiple testing methods scale up when both n and k increase?

Area of Science:

• Biopharmaceutical Research
• Genomic Data Analysis
• Statistical Methods

Background:

• Massive datasets in biopharmaceutical research, especially genomics, raise concerns about the scalability of multiple comparison methods.
• Traditional methods like familywise error rate control do not scale well with an increasing number of tests (k).
• False discovery rate controlling methods scale well with k, but not with increasing sample size (n) for point null hypotheses.

Purpose of the Study:

• To investigate the scale-up properties of various multiple comparison methods under increasing dataset sizes.
• To evaluate the performance of existing and novel statistical approaches in biopharmaceutical research contexts.
• To identify methods that best handle the challenges posed by large-scale genomic data analysis.

Main Methods:

• Development of a loss function approach to analyze the scalability of statistical methods.
• Comparison of different multiple comparison techniques, including Efron's recent proposal.
• Assessment of method performance with respect to the number of tests (k) and sample size (n).

Main Results:

• Familywise error rate methods exhibit poor scalability with increasing number of tests (k).
• False discovery rate methods demonstrate good scalability with increasing k, but not with increasing sample size (n).
• Efron's proposal shows the best scalability when both sample size (n) and number of tests (k) increase, though its performance can be inconsistent.

Conclusions:

• The choice of multiple comparison method significantly impacts the analysis of large-scale biopharmaceutical data.
• Efron's method offers advantages in scalability for high-dimensional genomic data but requires careful consideration of its potential erratic behavior.
• Further research is needed to refine methods for robust statistical inference in the era of big data in life sciences.