Testing for reviewer anchoring in peer review: A randomized controlled trial.

Area of Science:

• Scientific communication
• Research integrity

Background:

• The iterative peer review process involves initial reviews, author responses, and reviewer score updates.
• Anecdotal evidence suggests reviewers may not sufficiently adjust scores, potentially due to anchoring bias.

Purpose of the Study:

• To experimentally investigate whether peer reviewers exhibit anchoring bias to their initial scores.
• To determine if initial scores influence revised scores after authors address feedback.

Main Methods:

• A randomized controlled trial with 108 researchers was conducted.
• The experimental group reviewed a flawed paper version, then a corrected one; the control group reviewed only the correct version.
• Deception was employed to conceal the study's true purpose.

Main Results:

• The intervention successfully created differences in initial perceived paper quality between groups.
• No statistically significant evidence of anchoring bias was found when comparing revised scores to control group scores in either the Overall or Evaluation categories.

Conclusions:

• Reviewers did not demonstrate anchoring bias to their initial scores in this experimental setting.
• The findings suggest that the peer review update process may not be significantly hindered by score anchoring.