Can expected error costs justify testing a hypothesis at multiple alpha levels rather than searching for an elusive...

Area of Science:

• Statistics
• Statistical Inference
• Hypothesis Testing

Background:

• Conventional Neyman-Pearson framework allows hypothesis testing at a single alpha level.
• Researchers often face challenges in defining optimal alpha levels and error costs.
• Understanding error rates and strength of evidence is crucial in study design and reporting.

Purpose of the Study:

• To introduce and evaluate a multi-alpha level testing approach within the Neyman-Pearson framework.
• To demonstrate that multi-alpha level tests can achieve acceptable expected total error costs.
• To compare the performance of multi-alpha level tests with traditional single-alpha tests and optimization approaches.

Main Methods:

• Formulas for expected error costs were derived for both single and multiple alpha level tests.
• Prior probabilities of effect sizes were considered in both dichotomous and continuous distributions.
• Expected total costs were compared between single-alpha, multi-alpha, and optimal testing strategies.

Main Results:

• Multi-alpha level tests can yield acceptable expected total error costs.
• The sensitivity of optimization to error cost estimates and prevalence assumptions was highlighted.
• Testing at multiple default thresholds simplifies decision-making compared to formal optimization.

Conclusions:

• Multi-alpha level testing provides a practical alternative to single-alpha testing and complex optimization methods.
• While not strictly optimal, multi-alpha level tests may offer lower average error costs than methods relying on potentially mis-specified models.
• This approach encourages a more nuanced consideration of statistical error rates and evidence strength throughout the research process.