在大型语言模型BERT中分析参数结构结构的分析.
Pegah Ramezani1,2, Achim Schilling2,3, Patrick Krauss2,3
1Department of English and American Studies, University of Erlangen-Nuremberg, Erlangen, Germany.
Frontiers in artificial intelligence
|February 17, 2025
概括
伯特模型在层层地处理参数结构构造 (ASC),对象标记对于区分句型至关重要. 这揭示了变压器模型中的多层语言处理,反映了人类大脑的活动.
科学领域:
- 计算神经科学是一种神经科学.
- 认知科学 认知科学
- 自然语言处理自然语言处理.
背景情况:
- 了解大脑中的语言结构处理是认知神经科学的关键.
- 之前的研究分析了使用长短期记忆 (LSTM) 网络的参数结构构造 (ASC).
研究的目的:
- 在BERT语言模型中研究ASC处理和表示.
- 扩展以前在循环神经网络中对ASC的分析.
主要方法:
- 使用自定义GPT-4生成的数据集,包含四种ASC类型的2000个句子.
- 通过使用多维缩放 (MDS),t-SNE和通用歧视值 (GDV) 对12个层的BERT代币嵌入进行了分析.
- 训练有素的前分类器 (探头) 预测ASC类别,并使用费舍尔区分比 (FDR) 分析注意力权重.
主要成果:
- 在BERT 2-4层中,CLS代币嵌入显示出最好的ASC集群.
- 对象 (OBJ) 代币嵌入在从第10层集群中大幅增加,FDR得分高.
- 探测器准确度超过了90%的层2,表明隐藏的建筑信息不明显从GDV单独.
结论:
- 伯特表现出语言结构的复杂,分层处理,与LSTM有所不同.
- 对象和动词令牌对于区分BERT中的ASC至关重要.
- 变压器和循环模型显示了反映人类大脑语言处理的潜力.
关键词:
贝尔特 (BERT) 公司在 GPT-4 中使用.论证结构构造 论证结构构造计算语言学 计算语言学大型语言模型 (LLM)语言构造 (CXs) 是一种语言结构.自然语言处理 (NLP)句子表示的句子表示.更多相关视频
09:27Using Eye Movements Recorded in the Visual World Paradigm to Explore the Online Processing of Spoken Language
Published on: October 13, 2018
9.9K
05:22Dissociation of the Confounding Influences of Expectancy and Integrative Difficulty Residing in Anomalous Sentences in Event-related Potential Studies
Published on: May 9, 2019
5.3K
相关概念视频
Structure of Benzene: Kekulé Model
8.5K
In 1865, August Kekule suggested the structure of benzene according to the structural theory of organic chemistry based on the three assertions—formula of benzene is C6H6, all the hydrogens of benzene are equivalent, and each carbon must have four bonds due to its tetravalency.
He proposed that benzene has a cyclic structure of six carbon atoms attached to one hydrogen atom each, with three alternating pi bonds.
He proposed that benzene has a cyclic structure of six carbon atoms attached to one hydrogen atom each, with three alternating pi bonds.
8.5K
Structure of Benzene: Molecular Orbital Model
8.7K
According to the molecular orbital (MO) model, benzene has a planar structure with a regular hexagon of six sp2 hybridized carbons. As shown in Figure 1, each carbon is bonded to three other atoms with C–C–C and H–C–C bond angles of 120°. The C–H bond length is 109 pm, and the C–C bond length is 139 pm which is midway between the single bond length of sp3 hybridized carbons (154 pm) and sp2 hybridized carbons (133 pm).
8.7K
Components of Language
234
Language, whether spoken, signed, or written, consists of specific components: lexicon and grammar. The lexicon is the vocabulary of a language, comprising its words. Grammar is the set of rules used to convey meaning through the lexicon. For example, English grammar adds “-ed” to most verbs to indicate past tense. Words are formed by combining phonemes, which are the basic sound units of a language. Different languages have different sets of phonemes (e.g., “ah” vs.
234
Language and Cognition
317
Language serves as a bridge between ideas and communication, influencing how individuals perceive and interact with the world. Psychologists have long debated whether language shapes thought or vice versa. This discussion gained grip with Edward Sapir and Benjamin Lee Whorf in the 1940s, who proposed that language determines thought, a concept known as linguistic determinism. They suggested that the vocabulary and structure of a language influence how its speakers think and perceive reality.
317
Structuralism
577
Structuralism, an early psychological theory developed by Wilhelm Wundt and his student Edward Bradford Titchener, sought to dissect the human mind into its most fundamental components. Wundt's groundbreaking work in his laboratory set the stage for Titchener to define structuralism's goal as cataloging the "atoms" of the mind—sensations, images, and feelings—akin to how chemists identify elements of matter.
Titchener's approach to structuralism was unique. He...
Titchener's approach to structuralism was unique. He...
577
Structure of Alkanes
26.8K
The formation of carbon-carbon bonds leading to the creation of the carbon chain is the basis of organic chemistry. August Kekulé and Archibald Scott Couper independently developed this idea of carbon chain formation.
Hydrocarbons are the simplest organic compounds composed of carbons and hydrogens. Based on the bond order between carbons, the hydrocarbons are further classified into alkanes, alkenes, and alkynes.
Alkanes are the simplest hydrocarbons with sp3 hybrid carbon atoms....
Hydrocarbons are the simplest organic compounds composed of carbons and hydrogens. Based on the bond order between carbons, the hydrocarbons are further classified into alkanes, alkenes, and alkynes.
Alkanes are the simplest hydrocarbons with sp3 hybrid carbon atoms....
26.8K
