Systematic reviews of nonrandomized clinical studies in the orthopaedic literature

We systematically reviewed systematic reviews of surgical orthopaedic interventions published between 1996 and 2001 to document when and how nonrandomized studies were included. From more than 10,000 citations examined in various electronic databases, 58 orthopaedic systematic reviews were eligible for inclusion based on specific criteria. Thirty of these (52%) included nonrandomized studies, 15 of which found no randomized controlled trials. Systematic reviews were more likely to include randomized controlled trials if nondistinguishable operations were compared (if participants could be blinded). Only six of the systematic reviews that included nonrandomized studies (20%) assessed the quality of primary studies. Heterogeneity of studies was a major concern. In 21 of the systematic reviews that included nonrandomized studies (70%), data for groups treated similarly were pooled across studies, and outcomes for pooled groups were compared. The conclusions of systematic reviews that included nonrandomized studies are weakened by the limitations of nonrandomized study designs. The absence of established methods for including nonrandomized studies in systematic reviews, and consequently variability in the methods adopted, also limits the comparability of such reviews. Therefore the findings of systematic reviews that include nonrandomized studies should be interpreted with caution.