Approaches to analyzing binary data for large-scale A/B testing.

Area of Science:

• Statistics
• Experimental Design
• Industry Applications

Background:

• Industry standard uses t-tests for all A/B testing outcomes (continuous and binary).
• Naïve interim monitoring strategies are often applied without evaluating impact on power and type I error rates.
• Robustness of t-tests is known, but performance with large-scale binary A/B testing data and interim analyses requires specific evaluation.

Purpose of the Study:

• Evaluate statistical tests and study designs for large-scale industry A/B testing.
• Assess the t-test's performance on binary outcomes with and without interim analyses.
• Compare naïve interim monitoring with corrected approaches (e.g., O'Brien-Fleming).

Main Methods:

• Simulation studies comparing t-test, Chi-squared test, and Chi-squared test with Yate's correction for binary data.
• Evaluation of interim monitoring strategies: naïve approach vs. O'Brien-Fleming boundary.
• Consideration of early termination for futility, difference, or both.

Main Results:

• The t-test demonstrates comparable power and type I error rates for binary outcomes in large-scale industrial A/B tests.
• Performance is consistent with and without interim monitoring.
• Naïve interim monitoring without multiple testing corrections results in poorly performing studies.

Conclusions:

• The t-test is a robust choice for binary outcomes in large-scale industrial A/B testing.
• Interim analyses do not negatively impact the t-test's performance with large sample sizes.
• Corrected interim monitoring is crucial; naïve approaches should be avoided to maintain study integrity.