Empirical evidence of an association between internal validity and effect size in randomized controlled trials of...

We conducted a methodologic study.

The objective of this study was to assess the validity of the criteria list recommended by the Cochrane Back Review Group Editorial Board by evaluating whether individual items and a total score are associated with effect sizes in randomized controlled trials of back-pain interventions.

There is concern that studies of low methodologic quality may exaggerate the effectiveness of treatments for low back pain. We performed this study to examine the association between a common measure of internal validity and the reported magnitude of treatment effects.

We assessed the relationship between the 11 items contained in the Cochrane Back Review Group Internal Validity checklist and effect size in randomized trials of interventions for back pain. Of 267 trials in 15 Cochrane reviews that were eligible for inclusion, 51 were excluded, leaving 216 trials included in the analysis. The scores on the 11 items for each trial were taken from the original review. We extracted effect sizes from each low back pain trial.

We found that trials that fulfilled a specific item had smaller effect sizes compared with trials that did not fulfill that item for 10 of the 11 items, and for 6 of the criteria, the absolute difference in effect sizes was 0.10 or greater. The 95% confidence interval of the difference in effect sizes crossed the null value in each case. The number of items fulfilled showed that trials with higher scores consistently reported smaller effect sizes than trials with lower scores. At the thresholds of 5 or 6 items fulfilled, the difference in effect sizes was 0.20 in each case (95% confidence intervals 0.05-0.35 and 0.06-0.34, respectively). Stratified analyses did not support confounding by intervention.

We conclude that the 11-item Internal Validity Checklist is associated with effect size in randomized trials of interventions for back pain, and that our data support the use of a sum score of the number of fulfilled items in this list.