相关实验视频
Updated: Feb 4, 2026

07:13
A Two-interval Forced-choice Task for Multisensory Comparisons
Published on: November 9, 2018
11.5K
人类定生成AI的纵向比较与偏差校准的LLM-as-judge
1SUNY Empire State University, New York, United States of America.
PloS one
|February 2, 2026
概括
追踪大型语言模型 (LLM) 显示出不同的演变. 一些模型保持稳定,其他模型得到改善,一个模型退化,突出显示了可重复的LLM评估的挑战.
科学领域:
- 人工智能的人工智能
- 自然语言处理自然语言处理.
- 机器学习评估 机器学习评估
背景情况:
- 服务大型语言模型 (LLM) 在没有透明的变更日志的情况下迅速发展,阻碍了可重复的研究和评估.
- 评估LLM的性能和稳定性随着时间的推移至关重要,因为它们越来越多地融入各种应用.
研究的目的:
- 纵向跟踪主要服务LLM家族的性能和稳定性.
- 开发和验证一个可重复的LLM评估的强大方法,使用人类和LLM-as-judge评估.
- 识别模型漂移的模式,包括稳定性,改进和退化.
主要方法:
- 一个预先注册的,以人为的纵向研究设计,每周有十个评估波.
- 在六个不同的域中使用固定提示银行 (N=240) 进行一致的评估.
- 雇佣了盲目的人类评级者来判断正确性,以及一个偏差校准的LLM-as-judge来对准对偏好,每周使用布拉德利-特里模型进行校正.
- 应用混合效应建模和变化点检测 (PELT与MBIC罚款),以识别显著的漂移模式.
主要成果:
- 在三个主要的LLM家族中观察到不同的稳定轨迹:一个始终稳定,一个显示改善,一个显示研究中期退化.
- 法官校准提高了与人类判断的一致性 (肯德尔的t = 0.59-0.68) 并降低了评估波动性.
- 安全指标显示,与检测到的漂移事件存在共同变异,这表明行为转变,而不是确定的因果变化.
结论:
- 服务LLM具有独特和动态的稳定性模式,需要持续和可重复的评估框架.
- 开发的以人类为的LLM-as-judge方法提供了一个可扩展和可靠的方法来跟踪LLM演变.
- 在LLM开发和部署中的透明度对于培养信任和实现准确的绩效评估至关重要.
相关概念视频
The Anchoring-and-Adjustment Heuristic
7.8K
In order to make good decisions, we use our knowledge and our reasoning. Often, this knowledge and reasoning is sound and solid. However, sometimes, we are swayed by biases or by others manipulating a situation. For example, let’s say you and three friends wanted to rent a house and had a combined target budget of $1,600. The realtor shows you only very run-down houses for $1,600 and then shows you a very nice house for $2,000. Might you ask each person to pay more in rent to get the...
7.8K
Longitudinal Research
13.3K
Sometimes we want to see how people change over time, as in studies of human development and lifespan. When we test the same group of individuals repeatedly over an extended period of time, we are conducting longitudinal research. Longitudinal research is a research design in which data-gathering is administered repeatedly over an extended period of time. For example, we may survey a group of individuals about their dietary habits at age 20, retest them a decade later at age 30, and then again...
13.3K
Confirmation Biases
8.2K
The confirmation bias is the tendency to focus on information that confirms our existing beliefs and ignore information that is inconsistent with our expectations. For example, if you think that your professor is not very nice, you notice all of the instances of rude behavior exhibited by the professor while ignoring the countless pleasant interactions he is involved in on a daily basis. Have you ever fallen prey to the confirmation bias, either as the source or target of such bias?
8.2K
Hindsight Biases
4.3K
Hindsight bias leads you to believe that the event you just experienced was predictable, even though it really wasn’t. In other words, you knew all along that things would turn out the way they did. Can you relate this to the phrase "Hindsight is 20/20" now?
4.3K
Bias
7.4K
Bias refers to any tendency that prevents a question from being considered unprejudiced. In research, bias occurs when one outcome or answer is selected or encouraged over others in sampling or testing. Bias can occur during any research phase, including study design, data collection, analysis, and publication.
In statistics, a sampling bias is created when a sample is collected from a population, and some members of the population are not as likely to be chosen as others (remember, each member...
In statistics, a sampling bias is created when a sample is collected from a population, and some members of the population are not as likely to be chosen as others (remember, each member...
7.4K
Anchoring Junctions
5.0K
Anchoring junctions are multiprotein complexes that help cells connect to other cells and the extracellular matrix. Anchoring junctions are present on the lateral and basal surfaces of cells, providing strong and flexible connections. Focal adhesions are often formed due to cell interactions with the ECM substrata, which initiate signal transduction via kinase cascades and other mechanisms. Together, they provide stability and tissue integrity. There are three types of anchoring junctions:...
5.0K

