The frequentist implications of optional stopping on Bayesian hypothesis tests

Area of Science:

• Psychological research methodology
• Statistical inference
• Bayesian statistics

Background:

• Null hypothesis significance testing (NHST) is standard in psychology but often misapplied due to optional stopping.
• Bayesian hypothesis testing (BHT) is proposed as a solution, as Bayes factors are theoretically unaffected by stopping rules.

Purpose of the Study:

• To investigate the impact of optional stopping on Bayesian hypothesis testing (BHT) in common psychological research scenarios.
• To evaluate the frequentist properties of BHT when stopping rules are employed.

Main Methods:

• Quantitative analysis of optional stopping's effect on Bayes factors.
• Examination of two scenarios: composite hypotheses and heterogeneous populations.
• Assessment of BHT's frequentist implications under data-dependent stopping.

Main Results:

• Optional stopping can significantly influence Bayes factors in psychological research, contrary to theoretical expectations.
• The frequentist guarantees of BHT may not hold in practice due to common experimental stopping rules.
• Stopping rules can increase the likelihood of finding desired evidence, even when the Bayesian interpretation remains valid.

Conclusions:

• The frequentist implications of optional stopping on BHT require careful consideration in psychological research.
• Researchers should be aware of how stopping rules can bias evidence, even within a Bayesian framework.
• Methods to control for the impact of stopping rules on BHT are needed.