Teasing apart quality and validity in systematic reviews: an example from acupuncture trials in chronic neck and back...

The objectives of the study were (1) to carry out a systematic review to assess the analgesic efficacy and the adverse effects of acupuncture compared with placebo for back and neck pain and (2) to develop a new tool, the Oxford Pain Validity Scale (OPVS), to measure validity of findings from randomized controlled trials (RCTs), and to enable ranking of trial findings according to validity within qualitative reviews. Published RCTs (of acupuncture at both traditional and non-traditional points) were identified from systematic searching of bibliographic databases (e.g. MEDLINE) and reference lists of retrieved reports. Pain outcome data were extracted with preference given to standardized outcomes such as pain intensity. Information on adverse effects was also extracted. All included trials were scored using a five-item 0-16 point validity scale (OPVS). The individual RCTs were ranked according to their OPVS score to enable more weight to be placed on the trials of greater validity when drawing an overall conclusion about the efficacy of acupuncture for relieving neck and back pain. Statistical analyses were carried out on the OPVS scores to assess the relationship between trial finding (positive or negative) and validity. Thirteen RCTs met the inclusion criteria. Five trials concluded that acupuncture was effective, and eight concluded that it was not effective for relieving back or neck pain. There was no obvious difference between the findings of trials using traditional and non-traditional points. Using the new OPVS scale, the validity scores of the included trials ranged from 4 to 14. There was no significant relationship between OPVS score and trial finding (positive versus negative). Authors' conclusions did not always agree with their data. We drew our own conclusions (positive/negative) based on the data presented in the reports. Re-analysis using our conclusions showed a significant relationship between OPVS score and trial finding, with higher validity scores associated with negative findings. OPVS is a useful tool for assessing the validity of trials in qualitative reviews. With acupuncture for chronic back and neck pain, we found that the most valid trials tended to be negative. There is no convincing evidence for the analgesic efficacy of acupuncture for back or neck pain.