Analysis goals, error-cost sensitivity, and analysis hacking: Essential considerations in hypothesis testing and...

Area of Science:

• Statistics
• Research Methodology
• Scientific Inference

Background:

• The "replication crisis" in science is linked to incentives promoting selective reporting and misinterpretation of P-values and confidence intervals.
• Proposed solutions, like lowering significance thresholds (alpha levels) or using naive Bonferroni adjustments, are debated among statisticians.
• Existing approaches often fail to precisely define analysis goals, leading to controversies over appropriate statistical adjustments.

Purpose of the Study:

• To review issues in single-parameter inference, including error costs and loss functions, crucial for understanding statistical testing controversies.
• To examine arguments for and against modifying decision cut-offs and multiple comparison adjustments.
• To equip researchers with a better understanding of underlying assumptions in statistical inference debates.

Main Methods:

• Review of statistical inference principles, focusing on error costs and loss functions.
• Analysis of controversies surrounding significance testing and multiple comparison adjustments.
• Illustration of adjustment sensitivity to implicit decision costs using hypothesis testing scenarios.

Main Results:

• Statistical adjustment choices are highly sensitive to implicit decision costs, explaining disagreements among stakeholders.
• Justifying statistical decisions requires explicit cost functions, making inference controversies difficult to resolve without this.
• Pre-analysis statements of goals and plans can guide appropriate adjustments and counter inappropriate demands.

Conclusions:

• Understanding error costs and goal specification is essential for navigating statistical inference debates.
• Hierarchical (multilevel) regression methods offer a preferable alternative to conventional adjustments by integrating background information.
• Bayesian, semi-Bayes, and empirical-Bayes methods facilitate better-informed estimates for robust scientific decision-making.