The validity of level of evidence ratings of articles submitted to JBJS

In 2003, The Journal of Bone & Joint Surgery (American Volume) implemented a requirement for submitted clinical research articles to include a level of evidence rating. The aim of this study was to analyze the agreement between authors and JBJS regarding the level of evidence rating of accepted clinical articles.

A random sample of 353 clinical research articles accepted by JBJS from 2010 to 2012 was analyzed; 188 had a level of evidence rating provided by the author. Articles were grouped by study type and subspecialty. An unweighted kappa value was calculated to measure agreement between the authors and the JBJS editor, whose decision was used as the gold standard. In a secondary analysis, the articles in each subspecialty were categorized according to the year of submission to evaluate temporal trends.

Of the 353 articles, 69.4% (245) were classified by JBJS as representing a therapeutic study, 17.6% (sixty-two) were classified as representing Level-I evidence, and 25.2% (eighty-nine) dealt with arthroplasty. Agreement between the author and the JBJS editor was 0.79 (95% confidence interval [CI], 0.71 to 0.89; p < 0.001) for the study type, 0.62 (95% CI, 0.53 to 0.70; p < 0.001) for the level of evidence, and 0.65 (95% CI, 0.58 to 0.73; p < 0.001) for the full level of evidence rating (study type and level of evidence).

Level of evidence ratings suggested by authors from 2010 to 2012 had moderate to substantial agreement with the ratings assigned by the JBJS editor. This suggests that the level of evidence rating system is being properly understood by authors of articles published in JBJS. However, the low frequency of reporting suggests that JBJS needs to strictly enforce requirements for submission of a level of evidence rating at the time of manuscript submission.