Sequential hypothesis testing with Bayes factors: Efficiently testing mean differences.

Area of Science:

• Statistics
• Bayesian Inference
• Hypothesis Testing

Background:

• Optional stopping rules in Null Hypothesis Significance Testing (NHST) can inflate Type I error rates.
• Despite criticisms, researchers often continue data collection to achieve statistical significance.
• Bayesian hypothesis testing offers an alternative framework for flexible data collection strategies.

Purpose of the Study:

• To investigate the properties of a Sequential Bayes Factor (SBF) procedure for Bayesian hypothesis testing with optional stopping.
• To evaluate the long-term rate of misleading evidence, expected sample sizes, and effect size estimation bias.
• To compare the SBF design with optimal NHST for testing mean differences between two groups.

Main Methods:

• Developed and applied the Sequential Bayes Factor (SBF) procedure, computing Bayes factors sequentially until a predefined evidence level is met.
• Simulated data for testing mean differences between two groups under the SBF design.
• Analyzed long-term error rates, average sample sizes, and effect size bias.

Main Results:

• The SBF design allows flexible sampling plans without reliance on a priori power analyses.
• Compared to optimal NHST, the SBF design typically requires 50% to 70% smaller sample sizes.
• The SBF design demonstrated an equal or lower long-term rate of incorrect inferences.

Conclusions:

• Sequential Bayes Factors provide a statistically sound and efficient method for hypothesis testing with optional stopping.
• The SBF approach offers advantages in sample size reduction and maintaining inference accuracy.
• This Bayesian procedure addresses practical research needs for flexible data collection while ensuring robust conclusions.