斯托奇CA:一种利用预训练模型与交叉注意力的新方法
Seungwon Seo1, Suho Lee1, Sangheum Hwang2
1Department of Data Science, Seoul National University of Science and Technology, Seoul 01811, South Korea.
概括
我们介绍了随机交叉注意力 (StochCA),这是变压器模型的新型微调方法. 斯托奇CA增强了从预先训练的模型转移知识的性能,在转移学习和域泛化方面表现优于现有的方法.
科学领域:
- 人工智能的人工智能
- 机器学习 机器学习
- 深度学习 (Deep Learning) 是一种深度学习.
背景情况:
- 大规模预训练模型对于提高下游任务的性能至关重要.
- 标准微调可能无法充分利用这些预训练模型中的知识.
- 变压器架构被广泛使用,但需要有效的微调策略.
研究的目的:
- 为变压器模型引入一种新的微调方法,即随机交叉注意 (StochCA).
- 允许在微调过程中选择性地利用预训练模型的知识.
- 提高转移学习和域泛化任务的性能.
主要方法:
- 修改了变压器的自我注意机制,以结合交叉注意.
- 随机执行交叉注意力使用从预训练模型的相应块的键和值.
- 微调查询和目标模型的频道混合层,以利用预训练的表示.
主要成果:
- 随机交叉注意力 (StochCA) 在最先进的方法中表现出优越的性能.
- 在转移学习和域泛化基准中取得了显著的改进.
- 表明StochCA是对现有的微调方法的补充,可以将其结合起来以获得进一步的收益.
结论:
- 在变压器架构中,StochCA有效地增强了预训练模型的知识利用.
- 提出的方法为微调策略提供了显著的进步.
- 斯托奇CA提供了一种灵活而有效的方法,用于改善各种任务中的模型性能.
相关概念视频
Stereotype Content Model
14.0K
The Stereotype Content Model (SCM) was first proposed by Susan Fiske and her colleagues (Fiske, Cuddy, Glick & Xu, 2002; see also Fiske, 2012 and Fiske, 2017). The SCM specifies that when someone encounters a new group, they will stereotype them based on two metrics: warmth—or that group’s perceived intent, and how likely they are to provide help or inflict harm—and competence—or their ability to carry out that objective. Depending on the warmth-competence...
14.0K
Improving Translational Accuracy
2.5K
2.5K
Crossing Over
4.2K
Crossing over is the exchange of genetic information between homologous chromosomes during prophase I of meiosis I. Genetic recombination gives rise to allelic diversity in the newly formed daughter cells. In humans, crossing over produces genetically distinct haploid egg and sperm cells that undergo fertilization to produce unique offspring. Before cell division starts, the germ cell’s chromosome(s) undergo duplication in the S phase of the cell cycle. As the cells enter prophase I,...
4.2K
Test Cross
41.8K
Alleles are different forms of the same gene. Humans and other diploid organisms inherit two alleles of every gene, one from each parent.
41.8K
Crossover Experiments
2.7K
Crossover experiments, also called the repeated-measurements design, is a study design in which all experimental units are exposed to all treatments in different periods. Crossover experiments are generally used in psychology, the pharmaceutical industry, agriculture, and medicine.
Crossover designs are performed even with smaller sample sizes since the samples can act as their controls. These are better than simple randomized trials since patients are exposed to all the treatments.
Crossover designs are performed even with smaller sample sizes since the samples can act as their controls. These are better than simple randomized trials since patients are exposed to all the treatments.
2.7K
Cross-reactivity
31.0K
Overview
31.0K


