DDK:基于可微分搜索和BERT的递归知识蒸的动态结构修剪
Zhou Zhang1, Yang Lu2, Tengfei Wang1
1School of Computer Science and Information Engineering, Hefei University of Technology, Hefei 230009, China.
概括
我们开发了DDK,这是一种结合动态结构修剪和递归知识蒸的方法,用于压缩大型预训练语言模型,如BERT. 这种方法显著减少了实际NLP应用的计算需求.
科学领域:
- 人工智能的人工智能
- 自然语言处理自然语言处理.
- 机器学习 机器学习
背景情况:
- 大规模预训练模型 (例如BERT) 在NLP中表现出色,但需要大量的计算资源.
- 由于存储和处理需求,高参数数量阻碍了实际部署.
研究的目的:
- 为大型预训练语言模型提出一种新的压缩技术.
- 提高像BERT这样的模型的效率和部署性.
主要方法:
- 引入了DDK,一种结合动态结构修剪和递归知识蒸的方法.
- 利用可区分搜索来优化前层道和自我注意头.
- 实施递归知识蒸与适应权重,从中间层提取特征.
主要成果:
- 该DDK方法成功地压缩和加速了BERT模型.
- 与现有方法相比,GLUE基准标准的实验结果显示出更高的性能.
- 除分析验证了拟议的修剪和蒸策略的有效性.
结论:
- DDK 方法为压缩大型预训练语言模型提供了有效的解决方案.
- 这种方法解决了资源密集型NLP模型的实际部署挑战.
- DDK实现了最先进的性能,同时显著提高了模型效率.
相关概念视频
Survival Tree
85
Survival trees are a non-parametric method used in survival analysis to model the relationship between a set of covariates and the time until an event of interest occurs, often referred to as the "time-to-event" or "survival time." This method is particularly useful when dealing with censored data, where the event has not occurred for some individuals by the end of the study period, or when the exact time of the event is unknown.
Building a Survival Tree
Constructing a...
Building a Survival Tree
Constructing a...
85
Woodward–Hoffmann Selection Rules and Microscopic Reversibility
3.1K
Electrocyclic reactions, cycloadditions, and sigmatropic rearrangements are concerted pericyclic reactions that proceed via a cyclic transition state. These reactions are stereospecific and regioselective. The stereochemistry of the products depends on the symmetry characteristics of the interacting orbitals and the reaction conditions. Accordingly, pericyclic reactions are classified as either symmetry-allowed or symmetry-forbidden. Woodward and Hoffmann presented the selection criteria for...
3.1K
Deductive Reasoning
55.3K
Deductive reasoning, or deduction, is the type of logic used in hypothesis-based science. In deductive reasoning, the pattern of thinking moves in the opposite direction as compared to inductive reasoning, which means that it uses a general principle or law to predict specific results. From those general principles, a scientist can deduce and predict the specific results that would be valid as long as the general principles are valid.
For example, a researcher can deduce specific predictions...
For example, a researcher can deduce specific predictions...
55.3K
Piaget's Theory of Cognitive Development from Childhood into Adulthood
111
Jean Piaget's theory of cognitive development emphasizes the role of thinking in a child's learning process, suggesting that children are naturally curious about their environment. His approach to development is discontinuous, proposing that cognitive abilities progress through distinct stages, each with unique characteristics. Central to Piaget's theory is schemata—mental structures that allow individuals to understand and interpret the world.
Schemata: Building Blocks of...
Schemata: Building Blocks of...
111
Role of Shaping in Operant Conditioning
320
Shaping is a technique used in operant conditioning to train complex behaviors by rewarding successive approximations toward the target behavior. This method is necessary because organisms are unlikely to perform complex behaviors spontaneously. Instead, shaping breaks down the desired behavior into small, manageable steps.
The steps involved in shaping begin with reinforcing any response that resembles the desired behavior. For example, parents might praise a child for picking up one toy. As...
The steps involved in shaping begin with reinforcing any response that resembles the desired behavior. For example, parents might praise a child for picking up one toy. As...
320
Statically Indeterminate Problem Solving
378
Statically indeterminate problems are those where statics alone can not determine the internal forces or reactions. Consider a structure comprising two cylindrical rods made of steel and brass. These rods are joined at point B and restrained by rigid supports at points A and C. Now, the reactions at points A and C and the deflection at point B are to be determined. This rod structure is classified as statically indeterminate as the structure has more supports than are necessary for maintaining...
378


