衍生形态学揭示了大型语言模型中的模拟概括
Valentin Hofmann1,2, Leonie Weissweiler3, David R Mortensen4
1Allen Institute for AI, Seattle, WA 98103.
概括
像GPT-J这样的大型语言模型 (LLM) 使用类比,而不是规则,用于英语形容词命名中的语言概括. 这一发现表明,在LLM中,比以前理解的更重要的是模拟过程.
科学领域:
- 计算语言学 计算语言学
- 认知科学 认知科学
- 人工智能的人工智能
背景情况:
- 在大语言模型 (LLM) 中研究语言概括对于理解它们的能力至关重要.
- 之前的研究往往集中在正规的语言现象上,这使得基于规则的和模拟的学习难以区分.
- 在LLM语言概括中模拟推理的作用仍然未被充分探索.
研究的目的:
- 要确定在LLMs中的语言概括是否由规则或类比驱动.
- 检查英语形容词命名,一个可变的语言现象,以区分学习机制.
- 通过将认知模型与LLM预测进行比较,引入一种分析LLM概括的新方法.
主要方法:
- 专注于GPT-J模型来研究语言概括.
- 开发并将基于规则和模拟学习的认知模型与LLM培训数据相匹配.
- 将模型预测与GPT-J对nonce形容词的预测进行比较,特别是那些具有可变命名模式的形容词.
主要成果:
- 基于规则的模型和模拟模型都同样解释了GPT-J对正规形容词命名的预测.
- 模拟模型显著优于基于规则的模型,用于变量命名模式.
- 即使对于正规形式,GPT-J对单词频率的敏感性也与模拟账户保持一致.
结论:
- 这些发现驳斥了GPT-J在形容词命名中使用语言概括规则的假设.
- 证据强烈表明,模拟过程是GPT-J语言概括的基本机制.
- 这项研究强调了类比在大型语言模型语言能力中的重要作用.
相关概念视频
Generalization, Discrimination, and Extinction
383
Generalization, discrimination, and extinction are key concepts in operant conditioning that influence how behaviors are learned and maintained.
Generalization occurs when a behavior reinforced in one context is performed in similar situations. For instance, a student who studies diligently for calculus and receives excellent grades might apply the same study habits to psychology and history, expecting similar results. Generalization shows how learning in one setting can influence behavior in...
Generalization occurs when a behavior reinforced in one context is performed in similar situations. For instance, a student who studies diligently for calculus and receives excellent grades might apply the same study habits to psychology and history, expecting similar results. Generalization shows how learning in one setting can influence behavior in...
383
Language Development
293
Children master language quickly and with relative ease, supported by both biological predisposition and reinforcement. B. F. Skinner (1957) proposed that language is learned through reinforcement, while Noam Chomsky (1965) argued that language acquisition mechanisms are biologically determined.
The critical period for language acquisition suggests that the ability to acquire language is at its peak early in life. As people age, this proficiency decreases. Language development begins very...
The critical period for language acquisition suggests that the ability to acquire language is at its peak early in life. As people age, this proficiency decreases. Language development begins very...
293
Language and Cognition
303
Language serves as a bridge between ideas and communication, influencing how individuals perceive and interact with the world. Psychologists have long debated whether language shapes thought or vice versa. This discussion gained grip with Edward Sapir and Benjamin Lee Whorf in the 1940s, who proposed that language determines thought, a concept known as linguistic determinism. They suggested that the vocabulary and structure of a language influence how its speakers think and perceive reality.
303
Deductive Reasoning
54.8K
Deductive reasoning, or deduction, is the type of logic used in hypothesis-based science. In deductive reasoning, the pattern of thinking moves in the opposite direction as compared to inductive reasoning, which means that it uses a general principle or law to predict specific results. From those general principles, a scientist can deduce and predict the specific results that would be valid as long as the general principles are valid.
For example, a researcher can deduce specific predictions...
For example, a researcher can deduce specific predictions...
54.8K
Associative Learning
273
Associative learning is a fundamental concept in behavioral psychology, wherein a connection is established between two stimuli or events, leading to a learned response. This process is critical in understanding how behaviors are acquired and modified. Conditioning, the mechanism through which associations are formed, can be divided into two main types: classical conditioning and operant conditioning, each elucidating different aspects of associative learning.
Classical conditioning, also known...
Classical conditioning, also known...
273
Directional Terms
7.8K
Directional terms are essential for describing the relative locations of different body structures. For instance, an anatomist might describe one band of tissue as "inferior to" another, or a physician might describe a tumor as "superficial to" a deeper body structure. These terms often use comparative terms in pairs to trace out the relative locations of one body part to another or descriptions of body tissues like the deeper ones from superficially present with reference to...
7.8K


