The effect of dedicated methodology and statistical review on published manuscript quality

We examine how dedicated methodology and statistical review affects the quality of manuscripts published in Annals of Emergency Medicine.

The dedicated reviewers developed a manuscript scoring form based on previously used instruments. The form contained 84 unique elements. Eight items sought the presence of state-of-the-art features (eg, formal exploration of the sensitivity of results to assumptions); the others sought substandard quality. Two raters independently scored each original research publication appearing in 4 consecutive issues of Annals for the presence or absence of each relevant item. We then reviewed the formal methodology and statistical review and all subsequent correspondence between authors and editors to determine whether the methodology and statistical review provided guidance regarding each of the 84 items and whether the advice was incorporated into the final manuscript.

There were 32 original research articles. One was never subjected to methodology and statistical review. Reviewers agreed on 94% of all items; single-item agreement ranged from 77% to 100%. State-of-the-art features were present in 31 (14%) of 217 ratings; the methodology and statistical review had commented on 13 (42%) of these. State-of-the-art features were absent in 186 (86%) ratings; the methodology and statistical review had commented on 33 (18%) of these. Substandard features were deemed present in 166 (12%) of 1,519 ratings; the methodology and statistical review had commented on 82 (44%) of these. Substandard features were absent in 1,333 (88%) ratings; the methodology and statistical review had commented on 132 (10%) of these. We found no fatal flaws in the published manuscripts.

Methodology reviewers often failed to comment on deficiencies that they had classified as substandard when designing this study. Reviews also did not encourage inclusion of state-of-the-art abstract, article, and references features. When reviews identified areas in need of improvement, only half of the comments led to improved manuscripts. In the other half, authors either rebuked the suggestions or the editors did not act when suggestions were ignored.