无监督的句子表达学习与频率诱导的对抗调和不完整的句子过
Bing Wang1, Ximing Li1, Zhiyao Yang1
1College of Computer Science and Technology, Jilin University, China; Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education, Jilin University, China.
预训练的语言模型 (PLM) 由于单词频率而产生偏见的句子嵌入. 我们的 Slt-fai 框架通过使嵌入不变频率并强调具有信息性的低频词来改善无监督的句子表示学习.
科学领域:
- 自然语言处理自然语言处理.
- 机器学习 机器学习
背景情况:
- 预训练语言模型 (PLM) 是无监督句子表示学习 (USRL) 的基础.
- 由于单词频率的敏感性,PLM表现出异型嵌入空间,导致相似性和信息偏差,从而降低了句子嵌入质量.
研究的目的:
- 为了解决USRL中PLM的局限性.
- 提出一个新的框架,Slt-fai,通过减轻频率诱导的偏见来改善句子表示学习.
主要方法:
- 从PLM预培训机构和分配的频率标签中计算了单词频率.
- 开发了一个相似性歧视器,用于对抗性调,以创建频率不变嵌入.
- 引入了一个不完整的句子检测任务与信息区分器来强调有信息的低频词.
主要成果:
- 实现了一个均的频率不变的嵌入空间.
- 加强了对有信息性的低频词语的强调.
- 在各种骨干和数据集中证明了Slt-fai在现有的USRL基线上的优势.
结论:
- Slt-fai有效地克服了USRL的PLM中的频率诱导偏差.
- 该框架灵活,可插入和使用,并增强了句子嵌入质量.
- 对于无监督的句子表示学习,Slt-fai提供了显著的改进.
更多相关视频
05:54Eye-tracking to Distinguish Comprehension-based and Oculomotor-based Regressive Eye Movements During Reading
Published on: October 18, 2018
12:49Transcranial Direct Current Stimulation tDCS of Wernicke's and Broca's Areas in Studies of Language Learning and Word Acquisition
Published on: July 13, 2019
相关概念视频
Frequency-dependent Selection
Linear Approximation in Frequency Domain
In contrast, nonlinear systems do not inherently possess these properties. However, for small deviations around an operating point, a nonlinear system can often be approximated as linear....
Associative Learning
Classical conditioning, also known...
Determination of Expected Frequency
Construction of Frequency Distribution
First, make a table with two columns—one with the title of the data that needs to be organized, and the other column for frequency. [Draw a third column for tally marks if needed]. Then, take a look at the items given in the data set and decide if an ungrouped frequency distribution table or a grouped frequency distribution table would be more suitable. If there are large sets of different values, then it is...
Aliasing
If the sampling frequency is below the Nyquist rate, these replicas overlap, preventing the original...
