Evaluating the median p-value method for assessing the statistical significance of tests when using multiple...

Area of Science:

• Statistics
• Biostatistics
• Data Science

Background:

• Multiple imputation is a standard technique for handling missing data in statistical analyses.
• Rubin's Rules are widely used for pooling results from multiply imputed datasets but are not suitable for test statistics.
• Existing methods for pooling test statistics are complex and not widely implemented in software.

Purpose of the Study:

• To evaluate the performance of the median p-value method for pooling test statistics across imputed samples.
• To determine the reliability of the median p-value method for assessing statistical significance.

Main Methods:

• The median p-value method was tested with nine common statistical tests, including t-tests, ANOVA, correlation, and regression.
• Empirical type I error rates were calculated for each test under varying levels of missing data.
• Performance was compared against the advertised statistical significance levels.

Main Results:

• The median p-value method resulted in inflated empirical type I error rates for all tested statistical analyses.
• The inflation of type I error rates increased proportionally with the prevalence of missing data.
• The method demonstrated a consistent overestimation of statistical significance.

Conclusions:

• The median p-value method is unreliable for assessing statistical significance when pooling test statistics from imputed datasets.
• Researchers should avoid using the median p-value method due to its tendency to produce false positives.
• Alternative, more robust methods are needed for pooling test statistics in multiply imputed data.