SCORE Study Report 8: Closed Tests for All Pair-Wise Comparisons of Means

We compare five closed tests for strong control of family-wide type I error (FWE) while making all pair-wise comparisons of means in clinical trials with multiple arms such as the SCORE Study. We simulated outcomes of the SCORE Study under its design hypotheses, and used p-values from chi-squared tests to compare performance of a "pairwise" closed test described below to Bonferroni and Hochberg adjusted p-values. "Pairwise" closed testing was more powerful than Hochberg's method by several definitions of multiple-test power. Simulations over a wider parameter space, and considering other closed methods, confirmed this superiority for p-values based on normal, logistic, and Poisson distributions. The power benefit of "pair-wise" closed testing begins to disappear with 5 or more arms, and with unbalanced designs. For trials with 4 or fewer arms and balanced designs, investigators should consider using "pair-wise" closed testing in preference to Shaffer's, Hommel's, and Hochberg's approaches when making all pairwise comparisons of means. If not all p-values from the closed family are available, Shaffer's method is a good choice.