A large scale randomized controlled trial on herding in peer-review discussions.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Scientific Publishing

Background:

• Peer review is critical in academia, relying on human judgment susceptible to cognitive biases.
• Understanding and mitigating biases like herding behavior in peer review is essential for robust scientific evaluation.

Purpose of the Study:

• To investigate the presence of herding behavior in the discussion phase of academic peer review.
• To determine if the initial opinion presented in discussions disproportionately influences reviewer and discussion chair decisions.

Main Methods:

• A randomized controlled trial was conducted involving 1,544 papers and 2,797 reviewers at a top-tier machine learning conference.
• The study analyzed the conditional causal effect of the discussion initiator's opinion on paper outcomes.

Main Results:

• The experiment found no statistical evidence of herding behavior among reviewers or discussion chairs.
• This finding contrasts with previous research on anchoring effects and herding in other domains.

Conclusions:

• The absence of herding behavior suggests that current peer review discussion dynamics do not inherently lead to arbitrary decisions.
• The findings support the existing practice of not enforcing a unified policy for discussion initiation in peer review.