A permutational step-up method of testing multiple outcomes

This paper describes a permutational step-up multiple comparison procedure to adjust the p values from k related hypotheses. The method is applicable when two groups of subjects are being compared on each of k outcomes. It is related to the step-down method of Westfall and Young (1993, Resampling-Based Multiple Testing: Examples and Methods for P-Value Adjustment, Volume 1) and Troendle (1995, Journal of the American Statistical Association 90, 370-378) and is an alternative to the analytic step-up method of Dunnett and Tamhane (1992, Journal of the American Statistical Association 87, 162-170), which requires a specific distribution and correlation structure. By conditioning on the data observed, this permutational method avoids any specific distribution or correlation assumption. It is shown very generally that the method asymptotically controls the familywise probability of a type I error.