在A:在预先训练的语言模型上进行抑制适应
Cheng Kang1, Jindrich Prokop1, Lei Tong2
1Department of Cybernetics, Czech Technical University in Prague, Czech Republic.
概括
微调语言模型 (LMs) 可以通过抑制适应 (InA) 进行改进. 这种方法减少了可调节的重量,并消除了无关的知识,提高了下游任务的性能.
科学领域:
- 自然语言处理自然语言处理.
- 机器学习 机器学习
背景情况:
- 预先训练的语言模型 (LMs) 需要适应特定的下游任务.
- 现有的适应微调方法需要更清晰的机制和更好的信息传输控制.
研究的目的:
- 引入一种新的微调方法,抑制适应 (InA),用于预先训练的LMM.
- 为了减少可调节的重量和重权知识的数量,从预先训练的LMM.
主要方法:
- InA将可训练的向量插入到变压器注意力架构中.
- 使用值机制来消除不相关的知识,灵感来自于转抑制.
主要成果:
- 与现有的微调方法相比,InA实现了竞争力或更高的性能.
- 该方法在BERT-大,RoBERTa-大和DeBERTa-大模型上进行了评估.
- 对于文本分类和问答任务是有效的.
结论:
- 抑制适应 (InA) 为微调语言模型提供了一种高效有效的方法.
- 该方法提供了一种控制知识转移的机制,改善特定任务的性能.
相关概念视频
01:22Language Development
341
Children master language quickly and with relative ease, supported by both biological predisposition and reinforcement. B. F. Skinner (1957) proposed that language is learned through reinforcement, while Noam Chomsky (1965) argued that language acquisition mechanisms are biologically determined.
The critical period for language acquisition suggests that the ability to acquire language is at its peak early in life. As people age, this proficiency decreases. Language development begins very...
The critical period for language acquisition suggests that the ability to acquire language is at its peak early in life. As people age, this proficiency decreases. Language development begins very...
341
01:27Language and Cognition
342
Language serves as a bridge between ideas and communication, influencing how individuals perceive and interact with the world. Psychologists have long debated whether language shapes thought or vice versa. This discussion gained grip with Edward Sapir and Benjamin Lee Whorf in the 1940s, who proposed that language determines thought, a concept known as linguistic determinism. They suggested that the vocabulary and structure of a language influence how its speakers think and perceive reality.
342
02:34Enzymes
81.4K
Inside living organisms, enzymes act as catalysts for many biochemical reactions involved in cellular metabolism. The role of enzymes is to reduce the activation energies of biochemical reactions by forming complexes with its substrates. The lowering of activation energies favor an increase in the rates of biochemical reactions.
Enzyme deficiencies can often translate into life-threatening diseases. For example, a genetic abnormality resulting in the deficiency of the enzyme G6PD...
Enzyme deficiencies can often translate into life-threatening diseases. For example, a genetic abnormality resulting in the deficiency of the enzyme G6PD...
81.4K
01:52Eukaryotic Transcription Inhibitors
9.8K
Certain biochemical processes, such as embryonic development and cell growth regulation, depend on the repression of specific genes. DNA binding proteins known as eukaryotic transcription inhibitors regulate the repression of gene expression in eukaryotes. The presence of these inhibitors at the required location and time in the cell is triggered by the presence of hormones and additional signals from other cells.
Eukaryotic transcription inhibitors usually contain two distinct domains, a...
Eukaryotic transcription inhibitors usually contain two distinct domains, a...
9.8K
02:42Transcription Attenuation in Prokaryotes
15.3K
Transcriptional attenuation occurs when RNA transcription is prematurely terminated due to the formation of a terminator mRNA hairpin structure. Bacteria use these hairpins to regulate the transcription process and control the synthesis of several amino acids including histidine, lysine, threonine, and phenylalanine. Transcription attenuation takes place in the non-coding regions of mRNA.
There are several different mechanisms used to attenuate transcription. In ribosome mediated...
There are several different mechanisms used to attenuate transcription. In ribosome mediated...
15.3K