The 2005 meta-analysis of homeopathy: the importance of post-publication data

There is a discrepancy between the outcome of a meta-analysis published in 1997 of 89 trials of homeopathy by Linde et al and an analysis of 110 trials by Shang et al published in 2005, these reached opposite conclusions. Important data were not mentioned in Shang et al's paper, but only provided subsequently.

What was the outcome of Shang et al's predefined hypotheses? Were the homeopathic and conventional trials comparable? Was subgroup selection justified? The possible role of ineffective treatments. Was the conclusion about effect justified? Were essential data missing in the original article?

Analysis of post-publication data. Re-extraction and analysis of 21 higher quality trials selected by Shang et al with sensitivity analysis for the influence of single indications. Analysis of comparability. Sensitivity analysis of influence of subjective choices, like quality of single indications and of cut-off values for 'larger samples'.

The quality of trials of homeopathy was better than of conventional trials. Regarding smaller trials, homeopathy accounted for 14 out of 83 and conventional medicine 2 out of 78 good quality trials with n<100. There was selective inclusion of unpublished trials only for homeopathy. Quality was assessed differently from previous analyses. Selecting subgroups on sample size and quality caused incomplete matching of homeopathy and conventional trials. Cut-off values for larger trials differed between homeopathy and conventional medicine without plausible reason. Sensitivity analyses for the influence of heterogeneity and the cut-off value for 'larger higher quality studies' were missing. Homeopathy is not effective for muscle soreness after long distance running, OR=1.30 (95% CI 0.96-1.76). The subset of homeopathy trials on which the conclusion was based was heterogeneous, comprising 8 trials on 8 different indications, and was not matched on indication with those of conventional medicine. Essential data were missing in the original paper.

Re-analysis of Shang's post-publication data did not support the conclusion that homeopathy is a placebo effect. The conclusion that homeopathy is and that conventional is not a placebo effect was not based on comparative analysis and not justified because of heterogeneity and lack of sensitivity analysis. If we confine ourselves to the predefined hypotheses and the part of the analysis that is indeed comparative, the conclusion should be that quality of homeopathic trials is better than of conventional trials, for all trials (p=0.03) as well as for smaller trials (p=0.003).