Understanding unexpected results from randomized clinical trials - Does coffee reduce atrial fibrillation...

To explore the interpretation of unexpected results from a randomized controlled trial (RCT).

Adjunctive frequentist (power and type-M error) and Bayesian analyses were performed on a recently published RCT reporting a statistically significant relative risk reduction (p < 0.01) for caffeinated coffee drinkers compared with abstinence on atrial fibrillation (AF) recurrence. Individual patient data for the Bayesian survival models were reconstructed from the RCT published material and priors informed by the RCT power calculations.

The original RCT design had limited power for realistic effect sizes, increasing susceptibility to type-M (magnitude) error. Bayesian analyses also tempered the benefit for caffeinated coffee implied by standard statistical analysis resulting in only modest probabilities of clinically meaningful risk reductions (e.g., hazard ratio < 0.9 of 88% or a risk difference > 2% of 82%).

Supplemental frequentist and Bayesian approaches can provide robustness checks for unexpected RCT findings, providing contextualization, clarifying distinctions between statistical and clinical significance, and guiding replication needs.