Assessing the quality of randomized trials: reliability of the Jadad scale

An instrument was developed and validated by Jadad, et al. to assess the quality of clinical trials using studies from the pain literature. Our study determined the reliability of the Jadad scale and the effect of blinding on interrater agreement in another group of primary studies. Four raters independently assessed blinded and unblinded versions of 76 randomized trials. Interrater agreement was calculated among combinations of four raters for blinded and unblinded versions of the studies. A 4 x 2 x 2 repeated measures design was employed to evaluate the effect of blinding. The interrater agreement for the Jadad scale was poor (kappa 0.37 to 0.39), but agreement improved substantially (kappa 0.53 to 0.59) with removal of the third item (an explanation of withdrawals). Blinding did not significantly affect the Jadad scale scores. A more precise description of how to score the withdrawal item and careful conduct of a practice set of articles might improve interrater agreement. In contrast with the conclusions reached by Jadad, we were unable to demonstrate a significant effect of blinding on the quality scores.