On making multiple comparisons in clinical and experimental pharmacology and physiology

1. It is a central thesis of this review that in clinical and experimental pharmacology and physiology the goal of statistical analysis should be to minimize the risk of making any false-positive inferences from the results of an experiment (experimentwise Type I error). 2. It is common in clinical and experimental pharmacology and physiology for the effects of several treatments to be tested within a single experiment. Specific intercomparisons of these several effects, made in a pairwise or more complex fashion, inflates the risk of making false-positive inferences unless special statistical procedures are used. 3. A number of multiple comparison procedures is described and their ability to control experimentwise Type I error is evaluated critically. 4. When only a few (less than 5) of all possible pairwise or more complex comparisons are made between treatment groups, the Dunn-Sidák procedure provides maximum protection against excessive experimentwise Type I error and is very convenient to use. 5. When a control group is compared with all other treatment groups in a pairwise fashion, especially when the number of groups is large, the Dunnett procedure is more powerful than the Dunn-Sidák. 6. If investigators insist on making all possible pairwise comparisons among treatment groups, the Tukey-Kramer procedure provides maximum protection against false-positive inferences but inflates the Type II error rate. If it is especially important to avoid Type II error then the more complicated, stepwise procedures of the Ryan-Peritz-Welsch variety should be considered.