Reviewer bias in single- versus double-blind peer review

Area of Science:

• Computer Science
• Information Science
• Scholarly Communication

Background:

• Peer review is a cornerstone of academic publishing, with single-blind and double-blind being common models.
• Computer science research is frequently disseminated through peer-reviewed conferences, making their review processes critical.
• The Association for Computing Machinery International Conference on Web Search and Data Mining is a selective venue with a rigorous review process.

Purpose of the Study:

• To investigate the impact of single-blind versus double-blind peer review on paper evaluation in computer science conferences.
• To quantify potential biases introduced by author information in the peer review process.

Main Methods:

• A controlled experiment was conducted during the 10th Association for Computing Machinery International Conference on Web Search and Data Mining.
• Four committee members reviewed each paper: two with author information (single-blind) and two without (double-blind).
• Information asymmetry regarding author identity was maintained throughout the bidding, reviewing, and scoring stages.

Main Results:

• Single-blind reviewers bid on 22% fewer papers compared to double-blind reviewers.
• Reviewers in the single-blind condition showed a preference for papers from top universities and companies.
• Single-blind reviewers were significantly more likely to recommend papers from famous authors (OR=1.63), top universities (OR=1.58), and top companies (OR=2.10) for acceptance.

Conclusions:

• Single-blind peer review introduces tangible biases favoring well-known authors and institutions in computer science conference settings.
• The findings suggest that double-blind review may promote a more equitable evaluation of research submissions.
• This study highlights the importance of review methodology in maintaining fairness and objectivity in academic evaluations.