Relative likelihood ratios for neutral comparisons of statistical tests in simulation studies

Area of Science:

• Biostatistics
• Meta-analysis methodology

Background:

• Simulation studies often compare statistical tests using only statistical power.
• Comparing tests solely on power can be misleading if test sizes differ or deviate from nominal values.

Purpose of the Study:

• To introduce a novel approach for comparing multiple statistical tests by incorporating both statistical power and test size.
• To derive sample size formulas for comparative simulation studies using the new method.

Main Methods:

• Introduced relative positive and negative likelihood ratios, analogous to diagnostic accuracy metrics.
• Derived sample size formulas for comparative simulation studies.
• Applied the method to compare six statistical tests for small-study effects in meta-analyses.

Main Results:

• Demonstrated that comparing tests based on power alone, or adjusted/penalized for size, can yield misleading conclusions.
• The proposed likelihood ratio approach provides an accurate comparison of the power-size trade-off.

Conclusions:

• Relative likelihood ratios offer a robust framework for evaluating and comparing statistical tests in simulation studies.
• This method enhances the reliability of test selection in meta-analyses, particularly for assessing small-study effects.