通过有条件的在线知识转移促进微调
Zhiqiang Liu1, Yuhong Li1, Chengkai Huang2
1School of Software Engineering, South China University of Technology, China.
概括
条件在线知识传输 (COKT) 通过创建针对特定目标的规范化信号来提高有限数据的网络性能. 这种方法增强了微调,特别是在不相似的任务和小数据集.
科学领域:
- 机器学习 机器学习
- 计算机视觉 计算机视觉
- 深度学习 (Deep Learning) 是一种深度学习.
背景情况:
- 微调可以通过有限的标记数据来提高网络性能.
- 现有的方法使用源模型知识进行规范化,但可以引入偏差.
- 规范化信号通常独立于或松散地与目标信息相结合.
研究的目的:
- 提出一个有条件的在线知识转移 (COKT) 框架.
- 构建强大且与目标相关的规范化信号 (RS).
- 为了提高网络性能,在微调过程中改进知识传输.
主要方法:
- 开发了一个针对在线监管的目标主导的RS分支.
- 在RS分支中使用知识蒸.
- 通过样本明智的有条件注意力,残余特征融合和目标任务丢失集成的目标信息.
主要成果:
- 科克特显著超过了传统的微调基线.
- 不同目标任务和小数据集的性能增长值得注意.
- 该框架表现出强度和适应性.
结论:
- COKT有效地利用目标信息提供更准确的规范化信号.
- 拟议的方法提高了微调性能,特别是在具有挑战性的场景中.
- COKT可以扩展到跨模型和多模型微调设置.
更多相关视频
相关概念视频
Improving Translational Accuracy
11.3K
Base complementarity between the three base pairs of mRNA codon and the tRNA anticodon is not a failsafe mechanism. Inaccuracies can range from a single mismatch to no correct base pairing at all. The free energy difference between the correct and nearly correct base pairs can be as small as 3 kcal/ mol. With complementarity being the only proofreading step, the estimated error frequency would be one wrong amino acid in every 100 amino acids incorporated. However, error frequencies observed in...
11.3K
Observational Learning
188
Albert Bandura's observational learning, also known as imitation or modeling, occurs when a person observes and imitates another's behavior. It is a quicker process than operant conditioning. A well-known example is the Bobo doll study, where children who saw an adult acting aggressively towards the doll were more likely to act aggressively when left alone, compared to those who observed a nonaggressive adult. Many psychologists view observational learning as a form of latent learning...
188
Cognitive Learning
247
Cognitive learning is based on purposive behavior, incidental learning, and insight learning.
E. C. Tolman's theory of purposive behavior emphasizes that much behavior is goal-directed. He argued that to understand behavior, we must look at the entire sequence of actions leading to a goal. For instance, high school students study hard, not just due to past reinforcement but also to achieve the goal of getting into a good college.
Tolman introduced the idea that behavior is influenced by...
E. C. Tolman's theory of purposive behavior emphasizes that much behavior is goal-directed. He argued that to understand behavior, we must look at the entire sequence of actions leading to a goal. For instance, high school students study hard, not just due to past reinforcement but also to achieve the goal of getting into a good college.
Tolman introduced the idea that behavior is influenced by...
247
Purposive Learning
122
E. C. Tolman emphasized the purposiveness of behavior — the idea that much of our behavior is goal-directed. For instance, employees who aim for a promotion work diligently to meet their targets. Tolman argued that when classical conditioning and operant conditioning occur, the organism acquires certain expectations. In classical conditioning, a child might fear a dog because they expect it to bite. In operant conditioning, a person might consistently work overtime because they expect a...
122
Associative Learning
408
Associative learning is a fundamental concept in behavioral psychology, wherein a connection is established between two stimuli or events, leading to a learned response. This process is critical in understanding how behaviors are acquired and modified. Conditioning, the mechanism through which associations are formed, can be divided into two main types: classical conditioning and operant conditioning, each elucidating different aspects of associative learning.
Classical conditioning, also known...
Classical conditioning, also known...
408
The Anchoring-and-Adjustment Heuristic
7.3K
In order to make good decisions, we use our knowledge and our reasoning. Often, this knowledge and reasoning is sound and solid. However, sometimes, we are swayed by biases or by others manipulating a situation. For example, let’s say you and three friends wanted to rent a house and had a combined target budget of $1,600. The realtor shows you only very run-down houses for $1,600 and then shows you a very nice house for $2,000. Might you ask each person to pay more in rent to get the...
7.3K


