How do authors' perceptions of their papers compare with co-authors' perceptions and peer-review decisions?

Area of Science:

• Computer Science
• Scholarly Publishing
• Peer Review

Background:

• Understanding author perceptions versus actual outcomes in peer review is crucial for improving the scholarly publishing process.
• Previous studies have not extensively explored author self-assessment accuracy and its correlation with peer review outcomes in large-scale computer science conferences.

Purpose of the Study:

• To investigate the alignment between author predictions and actual acceptance rates in a major computer science conference.
• To analyze author self-ranking of scientific contribution and its relation to predicted acceptance and peer review decisions.
• To assess the impact of the peer review process on authors' perceptions of their own work.

Main Methods:

• Survey administered to over 23,000 authors submitting to NeurIPS 2021, covering predicted acceptance probability, self-perceived scientific contribution ranking, and post-review perception changes.
• Analysis of survey data against actual paper acceptance rates and peer review decisions.
• Statistical comparison of author perceptions based on demographics (gender) and reviewer status.

Main Results:

• Authors significantly overestimated acceptance probability (median 70% prediction vs. 25% actual rate).
• Female authors showed slightly higher miscalibration than male authors; invited reviewers were better calibrated than non-reviewers.
• Author rankings of contribution generally aligned with acceptance predictions (93%), but author-peer review agreement on rankings was only about 67%.
• Approximately 30% of authors reported improved perception of their work post-review, regardless of acceptance status.

Conclusions:

• Author self-assessment in high-stakes computer science conferences is prone to overestimation, highlighting a gap between author expectations and reality.
• Peer review decisions and author self-rankings show considerable disagreement, suggesting potential biases or differing evaluation criteria.
• The peer review process, even for rejected papers, can positively influence authors' views of their research, underscoring its value beyond acceptance decisions.