词时间序列中的长距离依赖:嵌入的等位数相关性
Paweł Wieczyński1, Łukasz Dębowski2
1Independent Researcher, 80-180 Gdańsk, Poland.
Entropy (Basel, Switzerland)
|June 26, 2025
概括
这项研究揭示了人类文本的远程依赖性 (LRD),但不是语言模型输出. 结果表明需要先进的,丰富内存的AI架构,超出当前的变压器模型.
科学领域:
- 自然语言处理自然语言处理.
- 计算语言学 计算语言学
- 信息理论 信息理论
背景情况:
- 长距离依赖 (LRD) 是具有记忆力的系统的特征,过去的事件在长时间内影响未来的结果.
- 在单词时间序列中分析LRD对于理解语言模式和开发复杂的语言模型至关重要.
- 传统模型,如隐藏的马尔科夫模型,难以有效地捕捉这些远程依赖关系.
研究的目的:
- 调查单词时间序列中远程依赖 (LRD) 的存在和特征.
- 为了比较人类生成文本中的LRD信号与大型语言模型 (LLM) 产生的文本.
- 为下一代LLM架构的开发提供信息.
主要方法:
- 利用从word2vec嵌入中获得的等号对应,作为Shannon相互信息的代理来量化LRD.
- 应用了Pinsker不等式,以建立一个理论上的联系之间的等号相关性和相互信息.
- 分析了标准化项目古堡集体 (人文) 和人与LLM文本集体 (LLM生成的文本).
主要成果:
- 检测到人文文本的协弦值相关性延伸指数衰减到大约1000个单词的滞后,证实LRD.
- 在大型语言模型生成的文本中没有观察到LRD的系统信号.
- 人类和LLM产生的物体之间的衰变模式有显著差异.
结论:
- 人类语言表现出长距离的依赖性,这种特征在当前的LLM生成文本中并不经常被发现.
- 现有的LLM架构,包括变压器,可能缺乏复制人类语言LRD所需的内存容量.
- 未来的LLM开发应该专注于新的,丰富记忆的架构,以更好地捕捉复杂的语言现象.
更多相关视频
06:48Lexical Decision Task for Studying Written Word Recognition in Adults with and without Dementia or Mild Cognitive Impairment
Published on: June 25, 2019
9.3K
09:09Foreign Accent and Forensic Speaker Identification in Voice Lineups: The Influence of Acoustic Features Based on Prosody
Published on: September 27, 2024
541
相关概念视频
Direction Cosines of a Vector
716
Direction cosines, which help describe the orientation of a vector with respect to the coordinate axes, are an essential concept in the field of vector calculus. Consider vector A that is expressed in terms of the Cartesian vector form using i, j, and k unit vectors. The magnitude of vector A is defined as the square root of the sum of the squares of its components. The direction of this vector with respect to the x, y, and z axes is defined by the coordinate direction angles α, β, and γ,...
716
¹H NMR: Long-Range Coupling
2.0K
The coupling interactions of nuclei across four or more bonds are usually weak, with J values less than 1 Hz. While these are usually not observed in spectra, the presence of multiple bonds along the coupling pathway can result in observable long-range coupling.
In alkenes, spin information is communicated via σ–π overlap, as seen in allylic (four-bond) and homoallylic (five-bond) couplings. These coupling interactions are stronger when the σ bond is parallel to the alkene...
In alkenes, spin information is communicated via σ–π overlap, as seen in allylic (four-bond) and homoallylic (five-bond) couplings. These coupling interactions are stronger when the σ bond is parallel to the alkene...
2.0K
Correlations
34.0K
Correlation means that there is a relationship between two or more variables (such as ice cream consumption and crime), but this relationship does not necessarily imply cause and effect. When two variables are correlated, it simply means that as one variable changes, so does the other. We can measure correlation by calculating a statistic known as a correlation coefficient. A correlation coefficient is a number from -1 to +1 that indicates the strength and direction of the relationship between...
34.0K
Correlation
12.5K
In statistics, two variables are said to be correlated if the values of one variable are associated with the other variable. Depending on the relationship between two variables, correlation can be of three types– positive correlation, negative correlation, and zero correlation.
Two variables, for example, a and b, are said to be positively correlated if both variables move in the same direction. In other words, a positive correlation exists between two variables, a and b, if:
Two variables, for example, a and b, are said to be positively correlated if both variables move in the same direction. In other words, a positive correlation exists between two variables, a and b, if:
12.5K
Coefficient of Correlation
6.4K
The correlation coefficient, r, developed by Karl Pearson in the early 1900s, is numerical and provides a measure of strength and direction of the linear association between the independent variable x and the dependent variable y.
If you suspect a linear relationship between x and y, then r can measure how strong the linear relationship is.
What the VALUE of r tells us:
The value of r is always between –1 and +1: –1 ≤ r ≤ 1.
The size of the correlation r indicates the...
If you suspect a linear relationship between x and y, then r can measure how strong the linear relationship is.
What the VALUE of r tells us:
The value of r is always between –1 and +1: –1 ≤ r ≤ 1.
The size of the correlation r indicates the...
6.4K
Drug Concentration Versus Time Correlation
1.3K
The plasma drug concentration-time curve is a crucial tool in pharmacokinetics, representing the drug's concentration in plasma at different time intervals post-administration. This curve illustrates the drug's journey from absorption into the systemic circulation, distribution to body tissues, and eventual elimination through excretion or biotransformation.
Two pivotal parameters are the minimum effective concentration (MEC) and the minimum toxic concentration (MTC). The MEC is the...
Two pivotal parameters are the minimum effective concentration (MEC) and the minimum toxic concentration (MTC). The MEC is the...
1.3K
