Replicability of Highly Cited Comparative Studies in Orthopedic Journals

Concerns are rising regarding the quality, validity, and reliability of clinical research findings in medical literature. This investigation sought to identify the most cited orthopedic clinical studies and assess the replicability of the findings reported.

Web of Science was used to identify the top ten orthopedic surgery journals by impact factor from which primary comparative studies cited at least 250 times were identified. A second literature search identified follow-up studies relevant to the respective primary studies. Studies investigating the same intervention via parallel methodology were summarized and their conclusions compared to their respective highly cited primary study.

Seven primary highly cited clinical studies met inclusion criteria. A literature search identified and screened 1163 follow-up articles, of which 79 met inclusion criteria. Of these, 70.9% (56/79) of studies were randomized clinical trials, 7.6% (6/79) were multicenter in nature, and 67% (53/79) were classified as level I evidence. Average subject cohort size in the follow-up studies was 365 patients (range, 10-4564). The rate of coming to the same conclusion as the primary study was 45.5% (36/79). The rate of different conclusions from the primary studies was 26.6% (21/79). Additionally, 16.5% (13/79) found a weaker correlation, and 11.4% (9/79) neither agreed nor disagreed with the primary study. No significant association existed between study design, level of evidence, or study size and agreement or disagreement with the original paper (P > .05).

Less than 50% of replicating follow-up studies support the effects demonstrated by highly cited comparative studies in orthopedic literature, which is a lower rate than that reported by other areas of medicine. Difficulty performing large, high-level-of-evidence studies and publication bias likely contribute to this observation. Based on these findings we believe that replication of prior research, emphasis on research quality, and conscious awareness of the limitations of clinical research are critical to the quality of orthopedic literature.