概念的几何学:稀疏的自动编码器特征结构
Yuxiao Li1, Eric J Michaud2,3, David D Baek3,4
1Beneficial AI Foundation (BAIF), Cambridge, MA 02139, USA.
Entropy (Basel, Switzerland)
|April 26, 2025
概括
大型语言模型 (LLM) 具有结构化的概念宇宙. 这项研究揭示了LLM特征表示中的原子,类似大脑和银河系规模的组织模式,为其内部工作提供了洞察力.
科学领域:
- 人工智能的人工智能
- 自然语言处理自然语言处理.
- 计算语言学 计算语言学
背景情况:
- 稀疏的自编码器可以将大型语言模型 (LLM) 的概念宇宙作为高维向量表示.
- 了解LLM的内部结构对于解释性和进一步发展至关重要.
研究的目的:
- 在LLMs中调查概念宇宙的多尺度结构组织.
- 识别和描述原子,中等和大规模的模式.
主要方法:
- 从稀疏的自编码器中获得的特征向量几何学的分析.
- 应用线性差别分析来预测分心方向.
- 使用多个指标量化空间模块化和特征集群的量化.
- 检查自值功率频谱和跨模型层的聚类.
主要成果:
- 在原子尺度上识别了具有平行四边形/梯形面的"晶体",通过分心投影进行了改进.
- 发现了"类似大脑"的模块化,在中间尺度上对特征 (如数学,代码) 进行空间聚类.
- 具有非同位素"星系"尺度结构的特征,具有功率定律自值分布,根据层次而异.
结论:
- 士课程概念空间展示了丰富的,多尺度的几何结构.
- 这种结构不是随机的,并且显示出类似于生物系统的模块化.
- 结果为理解和潜在地操纵LLM表示提供了一个框架.
相关概念视频
Natural and Artificial Concepts
80
In psychology, concepts can be divided into two categories: natural and artificial. Natural concepts are formed through direct or indirect experiences. For example, consider the concept of snow. If you live in a place with regular snowfall, such as Essex Junction, Vermont, you know snow through direct experiences. You’ve seen it fall, touched it, shoveled it, and played in it. You recognize its texture, appearance, and even its smell. In contrast, if you live on an island like Saint...
80
Vector Algebra: Method of Components
13.4K
It is cumbersome to find the magnitudes of vectors using the parallelogram rule or using the graphical method to perform mathematical operations like addition, subtraction, and multiplication. There are two ways to circumvent this algebraic complexity. One way is to draw the vectors to scale, as in navigation, and read approximate vector lengths and angles (directions) from the graphs. The other way is to use the method of components.
In many applications, the magnitudes and directions of...
In many applications, the magnitudes and directions of...
13.4K
Vector Algebra: Graphical Method
11.5K
Vectors can be multiplied by scalars, added to other vectors, or subtracted from other vectors. The vector sum of two (or more) vectors is called the resultant vector or, for short, the resultant.
We use the laws of geometry to construct resultant vectors, followed by trigonometry to find vector magnitudes and directions. For a geometric construction of the sum of two vectors in a plane, we follow the parallelogram rule. Suppose two vectors are at arbitrary positions. Translate either one of...
We use the laws of geometry to construct resultant vectors, followed by trigonometry to find vector magnitudes and directions. For a geometric construction of the sum of two vectors in a plane, we follow the parallelogram rule. Suppose two vectors are at arbitrary positions. Translate either one of...
11.5K
Cartesian Form for Vector Formulation
548
The Cartesian form for vector formulation is a process to calculate the moment of force using the position and force vectors. The moment of force is defined as the cross-product of these vectors, making it a vector quantity. The Cartesian form of the position and force vectors involves unit vectors, which can be used to express the cross-product in determinant form.
548
State Space Representation
145
The frequency-domain technique, commonly used in analyzing and designing feedback control systems, is effective for linear, time-invariant systems. However, it falls short when dealing with nonlinear, time-varying, and multiple-input multiple-output systems. The time-domain or state-space approach addresses these limitations by utilizing state variables to construct simultaneous, first-order differential equations, known as state equations, for an nth-order system.
Consider an RLC circuit, a...
Consider an RLC circuit, a...
145
Cartesian Vector Notation
677
Cartesian vector notation is a valuable tool in mechanical engineering for representing vectors in three-dimensional space, performing vector operations such as determining the gradient, divergence, and curl, and expressing physical quantities such as the displacement, velocity, acceleration, and force. By using Cartesian vector notation, engineers can more easily analyze and solve problems in various areas of mechanical engineering, including dynamics, kinematics, and fluid mechanics. This...
677


