在大型语言模型中反对语法封装的证据
Thomas A McGee1, Yiyang Zhang1, Idan A Blank1,2
1Department of Psychology, University of California, Los Angeles.
Cognitive science
|March 10, 2026
概括
大型语言模型 (LLM) 中的语法计算没有完全封装. 在BERT,GPT-2和Llama 2等模型中,语义合理性会影响语法专业化的注意点.
科学领域:
- 计算语言学计算语言学
- 人工智能的人工智能是人工智能.
- 认知科学是一种认知科学.
背景情况:
- 大型语言模型 (LLM) 使用专业头部的注意力机制来处理语言任务.
- 一个关键的问题是,这些头部的语法计算是否被封装或受其他信息类型的影响.
研究的目的:
- 调查语义可信性是否影响基于变压器的LLM中的语法专业注意点.
- 为了确定LLM中的语法计算是否被封装或可以透到语义信息中.
主要方法:
- 在三个LLM (BERT,GPT-2,Llama 2) 中确定了语法专业的注意点.
- 测试了语义可信度对这些专业头部内注意力权重的影响.
主要成果:
- 难以置信的语义信息减少了通过语法专业头关联的单词之间的注意力.
- 在几乎所有测试的依赖类型和模型中观察到这种效应.
结论:
- 在LLM注意力头中的语法信息是可以透到语义合理性.
- 这些发现表明了语法和语义的整合,反映了人类认知的方面.
更多相关视频
05:47Evidence-based Knowledge Synthesis and Hypothesis Validation: Navigating Biomedical Knowledge Bases via Explainable AI and Agentic Systems
Published on: June 13, 2025
1.8K
08:32Examining Online Syntactic Processing of Spoken Complex Sentences in Chinese Using Dual-Modal Interference Tasks
Published on: September 5, 2019
6.0K
相关概念视频
Components of Language
877
Language, whether spoken, signed, or written, consists of specific components: lexicon and grammar. The lexicon is the vocabulary of a language, comprising its words. Grammar is the set of rules used to convey meaning through the lexicon. For example, English grammar adds “-ed” to most verbs to indicate past tense. Words are formed by combining phonemes, which are the basic sound units of a language. Different languages have different sets of phonemes (e.g., “ah” vs.
877
Language and Cognition
912
Language serves as a bridge between ideas and communication, influencing how individuals perceive and interact with the world. Psychologists have long debated whether language shapes thought or vice versa. This discussion gained grip with Edward Sapir and Benjamin Lee Whorf in the 1940s, who proposed that language determines thought, a concept known as linguistic determinism. They suggested that the vocabulary and structure of a language influence how its speakers think and perceive reality.
912
Language Development
1.0K
Children master language quickly and with relative ease, supported by both biological predisposition and reinforcement. B. F. Skinner (1957) proposed that language is learned through reinforcement, while Noam Chomsky (1965) argued that language acquisition mechanisms are biologically determined.
The critical period for language acquisition suggests that the ability to acquire language is at its peak early in life. As people age, this proficiency decreases. Language development begins very...
The critical period for language acquisition suggests that the ability to acquire language is at its peak early in life. As people age, this proficiency decreases. Language development begins very...
1.0K
Language
974
Language is a unique communication system that uses words and systematic rules to organize and transmit information. Unlike other forms of communication, which may involve postures, movements, odors, or vocalizations, language relies on symbols and grammar. This makes human communication distinct from that of other species, who also communicate but do not use language in the same way humans do.
Corballis and Suddendorf (2007) and Tomasello and Rakoczy (2003) highlight the role of language in...
Corballis and Suddendorf (2007) and Tomasello and Rakoczy (2003) highlight the role of language in...
974
Constraints and Statical Determinacy
1.0K
In structural engineering, the equilibrium of a system is not only determined by its equations of equilibrium but also with the help of constraints. Constraints refer to restrictions on the motion of a system. The proper combinations of constraints can minimize the total number of constraints needed to maintain a system in mechanical equilibrium. When this happens, the system is said to be statically determinate. For such systems, the unknown reaction supports can be estimated using equilibrium...
1.0K
Higher Mental Functions of the Brain: Language
4.0K
Language is a system of communication that allows the expression of thoughts, ideas, and feelings. The brain processes language in both hemispheres.
Language formation and comprehension take place in the dominant hemisphere. The dominant hemisphere is responsible for understanding the meaning of spoken, written, or sign language, as well as the ability to communicate. For most people, the left hemisphere is the dominant one. The right hemisphere, then, gives tone and emotional context to the...
Language formation and comprehension take place in the dominant hemisphere. The dominant hemisphere is responsible for understanding the meaning of spoken, written, or sign language, as well as the ability to communicate. For most people, the left hemisphere is the dominant one. The right hemisphere, then, gives tone and emotional context to the...
4.0K
