Improving the quality of manuscript reviews: impact of introducing a structured electronic template to submit reviews

The purpose of this study is to evaluate the impact of structured electronic templates on the quality of manuscript reviews.

Twenty-five gastrointestinal and genitourinary reviewers for the American Journal of Roentgenology were included in this investigation. Reviewers were selected and anonymized on the basis of having reviewed one or more manuscripts during period 1 (January 2008 through December 2009). All manuscript reviews were given a review quality score. Reviewers with at least one suboptimal review (i.e., a quality review score of < 3 on a scale of 1 to 4, with 1 being poor and 4 being an excellent review) were selected for further follow-up. During period 1, the reviewers received minimal guidance regarding the expectations of a high-quality review. During period 2 (August 2010 through August 2011), the reviewers meeting the criteria selected for follow-up received a structured electronic template outlining the review process. Reviews were again scored for review quality and were compared with a paired Student t test.

The mean (± SD) and median review quality scores were 2.07 ± 0.44 and 2.0, respectively, for period 1 and 3.02 ± 0.89 and 3.0, respectively, for period 2. There was a 1-point improvement during period 2 after the introduction of the structured electronic template. Most of the reviews (19/25 [76%; 95% CI, 55%-91%]) improved after introduction of the structured electronic template, whereas only two of 19 worsened. Review scores significantly increased after introduction of the structured electronic templates (mean increase, 0.95 ± 0.92; t = 5.13; p < 0.0001). By specialty, the 13 gastrointestinal reviewers increased their score by 0.39 (p = 0.03), and the 12 genitourinary reviewers increased their score by 1.55 points (p < 0.0001).

The introduction of a structured electronic template significantly improved the quality of manuscript reviews submitted.