Estimation of false discovery rate using sequential permutation p-values

We consider the problem of testing each of m null hypotheses with a sequential permutation procedure in which the number of draws from the permutation distribution of each test statistic is a random variable. Each sequential permutation p-value has a null distribution that is nonuniform on a discrete support. We show how to use a collection of such p-values to estimate the number of true null hypotheses m0 among the m null hypotheses tested and how to estimate the false discovery rate (FDR) associated with p-value significance thresholds. We use real data analyses and simulation studies to evaluate and illustrate the performance of our proposed approach relative to standard, more computationally intensive strategies. We find that our sequential approach produces similar results with far less computational expense in a variety of scenarios.