A multiverse analysis of meta-analyses assessing acupuncture efficacy for smoking cessation evidenced vibration of...

To explore the impact of methodological choices on the results of meta-analyses (MAs), with acupuncture for smoking cessation as a case study.

After performing an umbrella review (using MEDLINE, the COCHRANE Library, the Wan Fang database, and the Chinese Journal Full-text Database/March 2018) of MAs exploring the use of acupuncture for smoking cessation, we extracted all randomized controlled trials. Numerous MAs were performed as per every possible combination of various methodological choices (e.g., characteristics of the intervention and control procedures, outcome, publication status, language) to assess their vibration of effects or more precisely the existence of a Janus effect, that is, whether the 10th and 90th percentiles in the distribution of effect sizes were in opposite directions.

After including 7 MAs and 39 randomized controlled trials, we performed 496,528 MAs. The effect size was negative at the 10th percentile (-0.1, favoring controls) and positive at the 90th percentile (1.17, favoring acupuncture). In all, 104,491 MAs showed a statistically significant difference in favor of acupuncture, whereas 392,037 failed to demonstrate the efficacy of acupuncture (including 96 that showed a statistically significant difference in favor of the control).

The methodological choices made in performing pairwise MAs can result in substantial vibration of effects, occasionally leading to opposite results.