层层的多式联络知识蒸用于视觉语言预训练模型
Jin Wang1, Dawei Liao1, You Zhang1
1School of Information Science and Engineering, Yunnan University, Kunming, China.
概括
知识蒸通过将知识层层地转移来压缩大型多式模式. 这种方法通过防止过度装配和模式干扰,提高了资源有限的设备的性能.
科学领域:
- 人工智能的人工智能
- 计算机视觉 计算机视觉
- 自然语言处理自然语言处理.
背景情况:
- 变压器模型通过处理图像和文本,在多式联运应用中表现出色.
- 大型模型尺寸对资源有限的设备的部署构成挑战,需要模型压缩技术,如知识蒸.
- 现有的多模式模型知识蒸方法往往导致学生模型过拟合,并与多模式干扰作斗争.
研究的目的:
- 为视觉语言预训练模型提出一种新的层级智能的多式联络知识蒸方法.
- 解决现有知识蒸技术的局限性,特别是过和模式干扰.
- 提高多式联运模式的效率和性能,以便在资源有限的设备上部署.
主要方法:
- 实现了层层的知识蒸,利用教师模型的中间层和最后层.
- 分开模式,并提供它们作为额外的输入,以减轻相互干扰.
- 引入了两个辅助损失,以提高每个模式的蒸效率.
主要成果:
- 提出的层级智慧多模式知识蒸显著超过现有的知识蒸方法.
- 在四个不同的多式联运任务中表现出卓越的性能.
- 成功地将知识从大型教师模型转移到较小的学生模型,而不会导致绩效下降.
结论:
- 层层的多模式知识蒸是压缩大型视觉语言预训练模型的有效策略.
- 拟议的方法减轻了过拟合和模式干扰,从而提高了学生模型的性能.
- 这种方法有助于在具有有限计算资源的设备上部署强大的多式联通AI.
相关概念视频
Improving Translational Accuracy
10.3K
Base complementarity between the three base pairs of mRNA codon and the tRNA anticodon is not a failsafe mechanism. Inaccuracies can range from a single mismatch to no correct base pairing at all. The free energy difference between the correct and nearly correct base pairs can be as small as 3 kcal/ mol. With complementarity being the only proofreading step, the estimated error frequency would be one wrong amino acid in every 100 amino acids incorporated. However, error frequencies observed in...
10.3K
Associative Learning
353
Associative learning is a fundamental concept in behavioral psychology, wherein a connection is established between two stimuli or events, leading to a learned response. This process is critical in understanding how behaviors are acquired and modified. Conditioning, the mechanism through which associations are formed, can be divided into two main types: classical conditioning and operant conditioning, each elucidating different aspects of associative learning.
Classical conditioning, also known...
Classical conditioning, also known...
353
Multicompartment Models: Overview
140
Multicompartment models are mathematical constructs that depict how drugs are distributed and eliminated within the body. They segment the body into several compartments, symbolizing various physiological or anatomical areas connected through drug transfer processes such as absorption, metabolism, distribution, and elimination.
These models offer a more comprehensive representation of drug behavior in the body than one-compartment models. They accommodate the complexity of drug distribution,...
These models offer a more comprehensive representation of drug behavior in the body than one-compartment models. They accommodate the complexity of drug distribution,...
140
Vision
53.2K
Vision is the result of light being detected and transduced into neural signals by the retina of the eye. This information is then further analyzed and interpreted by the brain. First, light enters the front of the eye and is focused by the cornea and lens onto the retina—a thin sheet of neural tissue lining the back of the eye. Because of refraction through the convex lens of the eye, images are projected onto the retina upside-down and reversed.
53.2K
Generalization, Discrimination, and Extinction
548
Generalization, discrimination, and extinction are key concepts in operant conditioning that influence how behaviors are learned and maintained.
Generalization occurs when a behavior reinforced in one context is performed in similar situations. For instance, a student who studies diligently for calculus and receives excellent grades might apply the same study habits to psychology and history, expecting similar results. Generalization shows how learning in one setting can influence behavior in...
Generalization occurs when a behavior reinforced in one context is performed in similar situations. For instance, a student who studies diligently for calculus and receives excellent grades might apply the same study habits to psychology and history, expecting similar results. Generalization shows how learning in one setting can influence behavior in...
548
Multi-input and Multi-variable systems
106
Cruise control systems in cars are designed as multi-input systems to maintain a driver's desired speed while compensating for external disturbances such as changes in terrain. The block diagram for a cruise control system typically includes two main inputs: the desired speed set by the driver and any external disturbances, such as the incline of the road. By adjusting the engine throttle, the system maintains the vehicle's speed as close to the desired value as possible.
In the absence...
In the absence...
106


