Further evaluating the conditional decision rule for comparing two independent means

Area of Science:

• Statistics
• Statistical Inference
• Hypothesis Testing

Background:

• Many statistical texts recommend a conditional decision rule for comparing independent group means.
• This rule dictates choosing between pooled variance and separate variance t tests based on a variance equality test.
• This approach is widely taught but its empirical performance is questioned.

Purpose of the Study:

• To empirically evaluate the Type I error rate of the conditional decision rule.
• To compare the conditional rule's performance against unconditional t tests and resampling methods.
• To assess performance across diverse distributions with varying skewness and kurtosis.

Main Methods:

• Empirical examination of Type I error rates.
• Comparison of four variance equality tests within the conditional decision rule framework.
• Evaluation against unconditional pooled variance t tests, separate variance Welch t tests, and resampling-based alternatives.
• Simulations using 49 distributions with varied skewness and kurtosis.

Main Results:

• The conditional decision rule did not consistently maintain the desired Type I error rate.
• Several unconditional tests, notably the separate variance t test, performed as well as or better than the conditional rule.
• The separate variance t test demonstrated robust performance across different distributional shapes.

Conclusions:

• The conditional decision rule for choosing between t tests is not empirically supported.
• Unconditional statistical tests, particularly the separate variance t test, are recommended for comparing independent group means.
• Abandoning the conditional decision rule in favor of unconditional approaches is advised for improved statistical practice.