在深度重复性语言模型中分析参数结构结构的分析
Pegah Ramezani1,2, Achim Schilling2,3, Patrick Krauss2,3
1Department of English and American Studies, University Erlangen-Nuremberg, Erlangen, Germany.
Frontiers in computational neuroscience
|July 1, 2025
概括
简单的循环神经网络,如长期短期记忆 (LSTM),可以学习参数结构构造 (ASC) 的抽象表示. 这支持语言处理中的基于预测的学习.
科学领域:
- 认知计算神经科学 认知计算神经科学
- 计算语言学 计算语言学
- 人工智能的人工智能
背景情况:
- 了解神经语言处理是认知神经科学的关键.
- 之前的工作分析了BERT中的参数结构构造 (ASCs).
- 这项研究研究了ASCs在一个更简单,大脑受约束的循环神经网络中.
研究的目的:
- 探索四个ASC在长短期记忆 (LSTM) 网络中的表示和处理.
- 为了确定更简单的神经架构是否可以形成抽象的,构建级别的表示.
- 为了支持这样的假设,分层语言结构是从基于预测的学习中产生的.
主要方法:
- 在2000个自定义GPT-4生成的句子上训练了一个LSTM.
- 使用多维缩放 (MDS) 和t-分布式静态邻居嵌入 (t-SNE) 分析了内部隐藏层激活.
- 使用通用歧视值 (GDV) 进行量化集群分离.
主要成果:
- 在所有LSTM隐藏层中观察到四个ASC的明显集群.
- ASC表示的最强的分离发生在最后一个隐藏层中.
- 结果与大型语言模型中以前的发现保持一致.
结论:
- 即使是简单的循环神经网络也可以开发语言结构的抽象表示.
- 结果支持通过基于预测的学习出现层次化的语言结构.
- 未来的工作将将模型表示与神经成像数据进行比较.
相关概念视频
Language and Cognition
456
Language serves as a bridge between ideas and communication, influencing how individuals perceive and interact with the world. Psychologists have long debated whether language shapes thought or vice versa. This discussion gained grip with Edward Sapir and Benjamin Lee Whorf in the 1940s, who proposed that language determines thought, a concept known as linguistic determinism. They suggested that the vocabulary and structure of a language influence how its speakers think and perceive reality.
456
Language Development
460
Children master language quickly and with relative ease, supported by both biological predisposition and reinforcement. B. F. Skinner (1957) proposed that language is learned through reinforcement, while Noam Chomsky (1965) argued that language acquisition mechanisms are biologically determined.
The critical period for language acquisition suggests that the ability to acquire language is at its peak early in life. As people age, this proficiency decreases. Language development begins very...
The critical period for language acquisition suggests that the ability to acquire language is at its peak early in life. As people age, this proficiency decreases. Language development begins very...
460
Components of Language
410
Language, whether spoken, signed, or written, consists of specific components: lexicon and grammar. The lexicon is the vocabulary of a language, comprising its words. Grammar is the set of rules used to convey meaning through the lexicon. For example, English grammar adds “-ed” to most verbs to indicate past tense. Words are formed by combining phonemes, which are the basic sound units of a language. Different languages have different sets of phonemes (e.g., “ah” vs.
410
Structuralism
1.2K
Structuralism, an early psychological theory developed by Wilhelm Wundt and his student Edward Bradford Titchener, sought to dissect the human mind into its most fundamental components. Wundt's groundbreaking work in his laboratory set the stage for Titchener to define structuralism's goal as cataloging the "atoms" of the mind—sensations, images, and feelings—akin to how chemists identify elements of matter.
Titchener's approach to structuralism was unique. He...
Titchener's approach to structuralism was unique. He...
1.2K
Elaborative Rehearsals
138
Elaborative rehearsal is a crucial cognitive strategy that strengthens information encoding in long-term memory by making meaningful connections between new data and pre-existing knowledge. This approach contrasts with maintenance rehearsal, which involves simple repetition without delving into the significance of the information. While maintenance rehearsal might temporarily keep information active in short-term memory, it is less effective for long-term retention.
The effectiveness of...
The effectiveness of...
138
Higher Mental Functions of the Brain: Language
1.0K
Language is a system of communication that allows the expression of thoughts, ideas, and feelings. The brain processes language in both hemispheres.
Language formation and comprehension take place in the dominant hemisphere. The dominant hemisphere is responsible for understanding the meaning of spoken, written, or sign language, as well as the ability to communicate. For most people, the left hemisphere is the dominant one. The right hemisphere, then, gives tone and emotional context to the...
Language formation and comprehension take place in the dominant hemisphere. The dominant hemisphere is responsible for understanding the meaning of spoken, written, or sign language, as well as the ability to communicate. For most people, the left hemisphere is the dominant one. The right hemisphere, then, gives tone and emotional context to the...
1.0K


