Power and Type I Error Control for Univariate Comparisons in Multivariate Two-Group Designs

Area of Science:

• Statistics
• Biostatistics
• Psychometrics

Background:

• Multiple-comparison procedures are crucial for controlling Type I errors in statistical analyses.
• Evaluating the performance of these procedures, particularly in two-group designs, is essential for accurate inference.
• Existing methods vary in their control of different error rates, such as familywise and per-family error rates.

Purpose of the Study:

• To evaluate the statistical power and Type I error control of various multiple-comparison procedures in two-group designs.
• To compare stepwise Bonferroni-based methods against other procedures concerning familywise Type I error rate (FWER) and per-family error rate (PFER).
• To highlight the importance of PFER control and identify methods that achieve it.

Main Methods:

• Computer simulations were employed to assess the performance of different multiple-comparison procedures.
• The simulations focused on two-group designs with varying numbers of outcome variables.
• Statistical power and Type I error rates (FWER and PFER) were the primary metrics evaluated.

Main Results:

• Stepwise Bonferroni-based procedures demonstrated higher statistical power but failed to control the PFER.
• Only the classical Bonferroni procedure and a modified MANOVA-protection method effectively controlled the PFER.
• The relative power of these two PFER-controlling methods was contingent upon several factors, including the number of outcome variables.

Conclusions:

• The choice of a multiple-comparison procedure is context-dependent.
• Factors influencing the decision include the number of outcome variables, the priority of PFER control, the need for confidence intervals, and the desired emphasis on multiple versus single variable significance.
• Greater attention to PFER is recommended, particularly for stepwise Bonferroni-type procedures.