t-tests, non-parametric tests, and large studies--a paradox of statistical practice?

Area of Science:

• Biostatistics
• Medical Research Methodology

Background:

• Increasing median sample sizes in high-impact medical journals over 30 years.
• Growing use of non-parametric tests at the expense of t-tests.
• Paradoxical trend of using non-parametric tests with increasing sample sizes.

Purpose of the Study:

• To explore the paradoxical practice of using non-parametric tests in large studies.
• To illustrate the consequences of using non-parametric tests with increasing sample sizes.
• To compare the performance of Wilcoxon-Mann-Whitney (WMW) and t-tests under various conditions.

Main Methods:

• Simulation study comparing rejection rates of WMW and two-sample t-tests.
• Samples drawn from skewed distributions with equal means/medians but differing spread.
• Analysis of test performance with increasing sample size, skewness, and spread differences.

Main Results:

• WMW test yields smaller p-values than t-tests, with discrepancies growing with sample size, skewness, and spread differences.
• For heavily skewed data (1000 observations/group, 10% std dev difference), WMW rejection rates can exceed 90%.
• High WMW rejection rates indicate power to detect differences in probability of one sample being less than another.

Conclusions:

• Non-parametric tests are best suited for small studies.
• Using non-parametric tests in large studies can lead to incorrect conclusions and reader confusion.
• T-tests and confidence intervals are recommended for large sample sizes, even with heavily skewed data.