Consistency between peer reviewers for a clinical specialty journal

Area of Science:

• Medical Journal Peer Review
• Scholarly Publishing
• Scientific Communication

Background:

• Assessing the consistency of peer review is crucial for maintaining journal quality.
• Unsolicited article evaluation relies heavily on reviewer recommendations for editorial decisions.
• Understanding reviewer agreement informs editorial processes and publication standards.

Purpose of the Study:

• To analyze the consistency between independent peer reviewers evaluating and ranking unsolicited articles.
• To investigate factors influencing reviewer agreement in scientific manuscript evaluation.
• To determine if editorial board membership affects peer reviewer consensus.

Main Methods:

• Analysis of paired reviews for 422 unsolicited submissions to the Journal of Clinical Anesthesia (1988-1991).
• Reviewer recommendations were categorized into four levels: accept outright, accept with revision, reject with resubmission, or reject outright.
• Statistical analysis (chi-square) was used to compare reviewer agreement based on editorial board membership.

Main Results:

• 40% of reviewer pairs showed agreement (consonant), and 40% differed by one category.
• 17% differed by two categories, and 3% differed by three categories.
• Articles reviewed by two editorial board members had lower agreement (32%) than those reviewed by two nonmembers (44%, p = .027).

Conclusions:

• Most peer reviews for unsolicited articles exhibit reasonable consonance or near-agreement.
• Editorial board members demonstrated significantly less agreement compared to non-members.
• Findings suggest potential implications of reviewer selection on the consistency of peer review outcomes.