规模问题:具有数十亿 (而不是数百万) 参数的大型语言模型更好地匹配自然语言的神经表征
Zhuoqiao Hong1, Haocheng Wang1, Zaid Zada1
1Department of Psychology and the Neuroscience Institute, Princeton University, Princeton, NJ.
bioRxiv : the preprint server for biology
|July 15, 2024
概括
较大的语言模型 (LLM) 更好地捕捉人类大脑.
科学领域:
- 计算神经科学是一种神经科学.
- 认知科学 认知科学
- 人工智能的人工智能
背景情况:
- 大型语言模型 (LLM) 显示了先进的语言处理能力.
- 神经科学研究在采用最新的LLM进展方面落后.
- 了解语言处理的神经基础从计算模型中受益.
研究的目的:
- 研究大语言模型 (LLM) 尺寸与它们在人类大脑中捕获语言信息的能力之间的关系.
- 从架构和训练数据大小中分离模型大小的影响.
- 探索LLM表示如何在自然语言理解过程中映射到神经活动.
主要方法:
- 利用了不同尺寸的基于变压器的多个LLM家族.
- 在患者中使用电皮质谱 (ECoG) 记录人类大脑活动.
- 配备了使用LLM上下文嵌入的电极智能编码模型,以预测自然主义故事倾听的神经信号.
主要成果:
- 较大的LLM在捕捉自然语言结构和预测神经活动方面表现出卓越的能力.
- 编码性能显示了与模型大小的日志线性关系,在早期的LLM层中达到顶峰.
- 预测的最佳层在大脑区域之间有所不同,这表明语言处理组织具有层次结构.
结论:
- 模型大小是LLM能够表示与人类大脑活动相关的语言信息的关键因素.
- 士学位的架构和训练数据与模型大小相互作用,影响神经编码.
- 研究结果支持使用越来越复杂的LLM来建模语言的神经基础.
相关概念视频
01:27Language and Cognition
340
Language serves as a bridge between ideas and communication, influencing how individuals perceive and interact with the world. Psychologists have long debated whether language shapes thought or vice versa. This discussion gained grip with Edward Sapir and Benjamin Lee Whorf in the 1940s, who proposed that language determines thought, a concept known as linguistic determinism. They suggested that the vocabulary and structure of a language influence how its speakers think and perceive reality.
340
01:26Scaling
243
In designing and analyzing filters, resonant circuits, or circuit analysis at large, working with standard element values like 1 ohm, 1 henry, or 1 farad can be convenient before scaling these values to more realistic figures. This approach is widely utilized by not employing realistic element values in numerous examples and problems; it simplifies mastering circuit analysis through convenient component values. The complexity of calculations is thereby reduced, with the understanding that...
243
01:29Mechanistic Models: Compartment Models in Algorithms for Numerical Problem Solving
48
Mechanistic models play a crucial role in algorithms for numerical problem-solving, particularly in nonlinear mixed effects modeling (NMEM). These models aim to minimize specific objective functions by evaluating various parameter estimates, leading to the development of systematic algorithms. In some cases, linearization techniques approximate the model using linear equations.
In individual population analyses, different algorithms are employed, such as Cauchy's method, which uses a...
In individual population analyses, different algorithms are employed, such as Cauchy's method, which uses a...
48