一些词比其他词更有价值吗?
1Center for Spoken Language Understanding, Oregon Health & Science University, Portland, Oregon, USA.
Proceedings of the Conference on Empirical Methods in Natural Language Processing. Conference on Empirical Methods in Natural Language Processing
|June 23, 2023
概括
新的评估指标提供了语言模型的更全面的评估,超出了简单的词精度. 这些指标揭示了性能差异,特别是低频词,解决诸如重复性文本生成等问题.
科学领域:
- 自然语言处理自然语言处理.
- 计算语言学 计算语言学
- 人工智能的人工智能
背景情况:
- 目前的语言模型评估指标主要集中在对地面真相的标记级准确性上.
- 像困惑度这样的传统指标可能会被语言的Zipfian分布所歪曲,过度强调频繁的单词.
- 现有的方法忽视了可能错误预测的单词的语言实用性,以及跨单词频率的模型性能变化.
研究的目的:
- 为语言模型引入新的内在评估措施.
- 为了提供一种比简单的准确性更全面的语言模型性能评估.
- 为了解决当前指标在捕捉不同单词频率的模型行为方面的局限性.
主要方法:
- 为单词预测任务开发两个新的内在评估指标.
- 使用拟议的指标对几个大型英语语言模型的评估.
- 将新指标的结果与传统评估方法进行比较.
主要成果:
- 拟议的指标揭示了语言模型之间的功能性能的差异.
- 当使用传统的评估指标时,这些差异并不明显.
- 这种新方法强调了在处理高频和低频单词方面的模型差异.
结论:
- 新的评估措施提供了对语言模型能力的更细致的理解.
- 这些指标可以识别潜在的故障模式,例如重复性文本生成.
- 这些发现表明需要在自然语言处理中使用更复杂的评估技术.
更多相关视频
06:33Decomposing the Variance in Reading Comprehension to Reveal the Unique and Common Effects of Language and Decoding
Published on: October 11, 2018
6.9K
09:20Cloud-Based Phrase Mining and Analysis of User-Defined Phrase-Category Association in Biomedical Publications
Published on: February 23, 2019
8.8K
相关概念视频
00:57Weighted Mean
5.2K
While taking the arithmetic, geometric, or harmonic mean of a sample data set, equal importance is assigned to all the data points. However, all the values may not always be equally important in some data sets. An intrinsic bias might make it more important to give more weightage to specific values over others.
For example, consider the number of goals scored in the matches of a tournament. While computing the average number of goals scored in the tournament, it may be more important to...
For example, consider the number of goals scored in the matches of a tournament. While computing the average number of goals scored in the tournament, it may be more important to...
5.2K
03:26Framing Effects
7.4K
Information is everywhere and its presentation—such as how and when items are presented—can impact our perceptions and decisions surrounding the info. This broad concept umbrellas framing effects—influences that occur due to the way information is framed in its appearance, whether it’s purely the order or the specific wording of a message. Let’s take a look at numerous ways in which two versions of something can objectively say the same thing, yet we respond in...
7.4K
00:54Ratio Level of Measurement
18.4K
The way a set of data is measured is called its level of measurement. Correct statistical procedures depend on a researcher being familiar with levels of measurement. For analysis, data are classified into four levels of measurement—nominal, ordinal, interval, and ratio.
A set of data measured using the ratio scale takes care of the ratio problem and provides complete information. Ratio scale data are like interval scale data, except they have a zero point and ratios can be calculated....
A set of data measured using the ratio scale takes care of the ratio problem and provides complete information. Ratio scale data are like interval scale data, except they have a zero point and ratios can be calculated....
18.4K
00:55Ordinal Level of Measurement
24.9K
The way a set of data is measured is called its level of measurement. Correct statistical procedures depend on a researcher being familiar with levels of measurement. For analysis, data are classified into four levels of measurement—nominal, ordinal, interval, and ratio.
Data measured using an ordinal scale are similar to nominal scale data, but there is one major difference. The ordinal scale data can be ordered. An example of ordinal scale data is a list of the top five national parks...
Data measured using an ordinal scale are similar to nominal scale data, but there is one major difference. The ordinal scale data can be ordered. An example of ordinal scale data is a list of the top five national parks...
24.9K
01:27Language and Cognition
377
Language serves as a bridge between ideas and communication, influencing how individuals perceive and interact with the world. Psychologists have long debated whether language shapes thought or vice versa. This discussion gained grip with Edward Sapir and Benjamin Lee Whorf in the 1940s, who proposed that language determines thought, a concept known as linguistic determinism. They suggested that the vocabulary and structure of a language influence how its speakers think and perceive reality.
377
01:23Primary and Secondary Reinforcers
309
In psychology, reinforcement is a key concept in behavior modification. B.F. Skinner demonstrated this with his experiments involving rats in what is known as a Skinner box. The rats learned to press a lever to receive food, a primary reinforcer that fulfilled their innate need for nourishment.
Effective reinforcers for humans vary depending on the individual and the context. Primary reinforcers, such as food, water, sleep, shelter, and pleasure, have inherent value and satisfy basic biological...
Effective reinforcers for humans vary depending on the individual and the context. Primary reinforcers, such as food, water, sleep, shelter, and pleasure, have inherent value and satisfy basic biological...
309