Evaluating the surgery literature: can standardizing peer-review today predict manuscript impact tomorrow?

Evidence-based surgery is predicated on the quality of published literature. We measured the quality of surgery manuscripts selected by peer review and identified predictors of excellence.

One hundred twenty clinical surgery manuscripts were randomly selected from 1998 in 5 eminent peer-reviewed surgery and medical journals. Manuscripts were blinded for author, institution, and journal of origin. Four surgeons and 4 methodologists evaluated the quality using novel instruments based on subject selection, study protocol, statistical analysis/inference, intervention description, outcome assessments, and results presentation. Predictors of quality and impact factor were identified using bivariate and multivariate regression.

Oncology was the most common subject (26%), followed by general surgery/gastrointestinal (24%). The average number of study subjects was 417; the majority of manuscripts were American (53%), from a single institution (59%). Eighteen percent had a statistician author. Mean number of citations was 128. Surgery manuscripts from medical, compared with surgery journals, had better total quality scores (3.8 vs. 5.2, P < 0.001). They had more subjects and were more likely to have a statistician as coauthor (43% vs. 10%, P < 0.001), multi-institutional, international collaboration (30% vs. 8%, P < 0.001), and higher citation index (mean: 350 vs. 54, P < 0.001). They were more often foreign (70% vs. 40%, P < 0.001). Independent predictors of quality were having a statistician coauthor, study funding, European origin, and more study subjects. Quality assessment using our instruments predicted the number of citations after 10 years (P < 0.01), along with having a statistician coauthor, international multi-institutional collaboration, and more subjects.

The quality of surgery manuscripts can be improved by including a statistician as coauthor, with efforts directed toward implementing multi-institutional/interdisciplinary trials. Peer-review across journals can be standardized through the use of instruments measuring methodologic and clinical quality.