同行评价的同行评价:一个随机对照试验和其他实验
Alexander Goldberg1, Ivan Stelmakh2, Kyunghyun Cho3
1School of Computer Science, Carnegie Mellon University, Pittsburgh, Pennsylvania, United States of America.
PloS one
|April 2, 2025
概括
评估同行评审质量是一项挑战. 这项研究发现,评论长度和作者偏见显著影响质量评估,反映了论文评论中发现的问题.
科学领域:
- 人工智能的人工智能
- 机器学习 机器学习
- 科学出版科学出版
背景情况:
- 评估同行评审质量对于激励高质量的贡献和实验研究至关重要.
- 现有研究强调了评估同行评审质量的挑战.
研究的目的:
- 调查同行评审质量评估的可靠性.
- 在对同行评审的评估中识别偏见和不一致.
主要方法:
- 在NeurIPS 2022上进行的一项大规模研究,涉及到meta-reviewer和作者评估提交的论文评论.
- 一个随机对照试验操纵审查长度来测试偏差.
- 对观察数据的分析,以评估作者偏见和评估者之间的协议.
主要成果:
- 人工延长的评论在质量方面被评为明显更高.
- 作者表现出对评论的积极偏见,建议接受他们的论文.
- 在审查评估者中观察到高分歧率 (28-32%) 和显著的校准错误.
结论:
- 同行评价的评估也存在类似于论文评价的问题,包括不一致性,偏见,错误校准和主观性.
- 可靠地评估同行评审质量仍然是一个重大挑战.
- 调查结果表明,需要改进评估审查质量的方法.
相关概念视频
Social Proof
27.3K
Social proof is a form of persuasion based on comparison and conformity. People compare their behavior and actions to what others are doing and will change to conform to do what their peers do.
27.3K
Naturalistic Observations
15.3K
If you want to understand how behavior occurs, one of the best ways to gain information is to simply observe the behavior in its natural context. However, people might change their behavior in unexpected ways if they know they are being observed. How do researchers obtain accurate information when people tend to hide their natural behavior? As an example, imagine that your professor asks everyone in your class to raise their hand if they always wash their hands after using the restroom. Chances...
15.3K
Self-Evaluation: Self-Enhancement and Self-Verification
5.1K
Social psychologists have documented that feeling good about ourselves and maintaining positive self-esteem is a powerful motivator of human behavior (Tavris & Aronson, 2008). In the United States, members of the predominant culture typically think very highly of themselves and view themselves as good people who are above average on many desirable traits (Ehrlinger, Gilovich, & Ross, 2005). Often, our behavior, attitudes, and beliefs are affected when we experience a threat to our...
5.1K
Ethics in Research
22.9K
Today, scientists agree that good research is ethical in nature and is guided by a basic respect for human dignity and safety. However, this has not always been the case. Modern researchers must demonstrate that the research they perform is ethically sound.
22.9K
Group Design
8.8K
The most basic experimental design involves two groups: the experimental group and the control group. The two groups are designed to be the same except for one difference— experimental manipulation. The experimental group gets the experimental manipulation—that is, the treatment or variable being tested—and the control group does not. Since experimental manipulation is the only difference between the experimental and control groups, we can be sure that any differences between...
8.8K
Blind Procedures
10.6K
Ideally, the people who observe and record the children’s behavior are unaware of who was assigned to the experimental or control group, in order to control for experimenter bias. Experimenter bias refers to the possibility that a researcher’s expectations might skew the results of the study. Remember, conducting an experiment requires a lot of planning, and the people involved in the research project have a vested interest in supporting their hypotheses. If the observers knew which...
10.6K


