大型语言模型中的事件知识:不可能与不可能之间的差距.
Carina Kauf1,2, Anna A Ivanova1,2,3, Giulia Rambelli4
1Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology.
Cognitive science
|November 27, 2023
概括
大型语言模型 (LLM) 通过区分可能的和不可能的场景来证明重要的事件知识. 然而,他们对可能和不可能事件的理解仍然不那么一致,这表明他们对一般事件的理解存在差距.
科学领域:
- 计算语言学 计算语言学
- 认知科学 认知科学
- 人工智能的人工智能
背景情况:
- 语言体中的单词共同出现模式编码概念知识.
- 大型语言模型 (LLM) 利用这些模式进行语义任务和获取世界知识.
- 没有研究过LLM对常见事件的概括知识的程度.
研究的目的:
- 调查预先训练有素的LLM是否能够区分可信和不可信的代理人-患者相互作用描述.
- 评估经过分布式语言模式培训的法学士所获得的事件知识的程度.
- 在理解事件语义方面,将LLM性能与其他分布式语言模型进行比较.
主要方法:
- 测试了五个预训练的LLM (BERT到MPT) 在精选的最小句子对 (n=1215) 上.
- 比较了可信与不可信事件描述的LLM概率赋值.
- 分析了影响LLM成绩的因素,包括表面层面的特征和语法/语义变体.
主要成果:
- LLM 始终将可能发生的事件的概率高于不可能发生的事件 (例如",老师买了笔记本电脑"和"笔记本电脑买了老师").
- LLM对可能而不是不太可能的事件的偏好不太一致 (例如",保姆辅导男孩"和"男孩辅导保姆").
- 法学士的表现受到句子可信性和表面特征的影响,在语法上比语义上更好地概括.
结论:
- 预训练有素的LLM从分布式语言模式中获得实质性的事件知识,特别是区分可能的和不可能的事件.
- 有一个显著的差距存在于LLMs在持续区分可能和不太可能的事件的能力.
- 句子可信性是LLM内部表示的一个组织维度,但细微的事件理解需要进一步发展.
更多相关视频
05:47Evidence-based Knowledge Synthesis and Hypothesis Validation: Navigating Biomedical Knowledge Bases via Explainable AI and Agentic Systems
Published on: June 13, 2025
232
05:22Dissociation of the Confounding Influences of Expectancy and Integrative Difficulty Residing in Anomalous Sentences in Event-related Potential Studies
Published on: May 9, 2019
5.4K
相关概念视频
Unusual Results
3.2K
Unusual results are those that have a very low chance of occurring. Unusual results can be identified using probabilities and the range rule of thumb. In problems involving probability, unusual results can be observed in 2 instances – an unusually high number of successes or an unusually low number of successes.
According to the range rule of thumb, any value above or below two standard deviations, 2σ from the mean, μ is considered unusual.
Maximum unusual value =...
According to the range rule of thumb, any value above or below two standard deviations, 2σ from the mean, μ is considered unusual.
Maximum unusual value =...
3.2K
Probability Laws
40.9K
Overview
40.9K
Uncertainty: Overview
563
In analytical chemistry, we often perform repetitive measurements to detect and minimize inaccuracies caused by both determinate and indeterminate errors. Despite the cares we take, the presence of random errors means that repeated measurements almost never have exactly the same magnitude. The collective difference between these measurements - observed values - and the estimated or expected value is called uncertainty. Uncertainty is conventionally written after the estimated or expected value.
563
Propagation of Uncertainty from Random Error
699
An experiment often consists of more than a single step. In this case, measurements at each step give rise to uncertainty. Because the measurements occur in successive steps, the uncertainty in one step necessarily contributes to that in the subsequent step. As we perform statistical analysis on these types of experiments, we must learn to account for the propagation of uncertainty from one step to the next. The propagation of uncertainty depends on the type of arithmetic operation performed on...
699
Generalization, Discrimination, and Extinction
572
Generalization, discrimination, and extinction are key concepts in operant conditioning that influence how behaviors are learned and maintained.
Generalization occurs when a behavior reinforced in one context is performed in similar situations. For instance, a student who studies diligently for calculus and receives excellent grades might apply the same study habits to psychology and history, expecting similar results. Generalization shows how learning in one setting can influence behavior in...
Generalization occurs when a behavior reinforced in one context is performed in similar situations. For instance, a student who studies diligently for calculus and receives excellent grades might apply the same study habits to psychology and history, expecting similar results. Generalization shows how learning in one setting can influence behavior in...
572
Hindsight Biases
3.4K
Hindsight bias leads you to believe that the event you just experienced was predictable, even though it really wasn’t. In other words, you knew all along that things would turn out the way they did. Can you relate this to the phrase "Hindsight is 20/20" now?
3.4K
