Large-scale language analysis of peer review reports

Area of Science:

• Bibliometrics
• Scholarly Communication
• Scientific Publishing

Background:

• Peer review is crucial for scientific publishing but faces criticism for subjectivity and bias.
• Limited access to peer review reports has historically hindered empirical research into its linguistic characteristics.

Purpose of the Study:

• To investigate the linguistic characteristics of a large corpus of peer review reports.
• To determine the influence of reviewer recommendation, research area, peer review type, and reviewer gender on these linguistic features.

Main Methods:

• Utilized text-analysis software to analyze 472,449 peer review reports.
• Quantified linguistic characteristics including analytical tone, authenticity, clout, sentiment, and morality.
• Examined these characteristics in relation to reviewer recommendation, research area, peer review type, and reviewer gender.

Main Results:

• Reviewer recommendation was the most significant factor influencing the linguistic characteristics of peer review reports.
• Research area, type of peer review, and reviewer gender demonstrated minimal to no impact on report linguistics.
• The consistency across different research areas, review types, and reviewer genders suggests a uniform application of review criteria.

Conclusions:

• The linguistic characteristics of peer review reports are primarily driven by the reviewer's recommendation.
• The minimal influence of extraneous factors like research area, review type, or reviewer gender suggests the inherent robustness and standardization of the peer review process.
• Findings support the reliability of peer review as a quality control mechanism in scientific publishing.