Structured peer review: pilot results from 23 Elsevier journals

Area of Science:

• Scientific publishing
• Scholarly communication
• Peer review processes

Background:

• Traditional peer review lacks consistency, hindering manuscript and review quality assessment.
• A pilot study was conducted to evaluate the implementation of structured peer review.
• The study aimed to explore reviewer engagement, agreement levels, and refine structured questions.

Purpose of the Study:

• To assess reviewer adaptation to a structured peer review format.
• To analyze reviewer agreement on specific manuscript aspects.
• To compare agreement rates before and after structured peer review implementation.

Main Methods:

• A structured peer review format with nine questions was piloted across 220 journals.
• 107 manuscripts from 23 journals, with two review reports each, were analyzed.
• Qualitative methods were used to independently analyze reviewer answers by two raters.

Main Results:

• Reviewers (92%) answered structured questions, with the most detailed responses concerning methods for reproducibility.
• Highest agreement was on manuscript flow (72%), lowest on data interpretation (53%) and statistical analysis (52%).
• Absolute agreement on final recommendations increased to 41% from 31% (p=0.0275) compared to pre-implementation.

Conclusions:

• Reviewers adapted to structured peer review, covering more topics than in traditional reports.
• Disagreements persist in interpreting results and statistical analysis quality.
• Structured peer review may improve recommendation agreement, but further research is needed to confirm benefits.