噪音监督在基金会学习模式学习中的影响
概括
这项研究揭示了基础模型预训练数据集中的噪音可以损害域外性能. 一种新的调方法NMTune有效地减轻了这种噪音,改善了噪音模型转移学习的模型概括.
科学领域:
- 深度学习 (Deep Learning) 是一种深度学习.
- 机器学习 机器学习
- 人工智能的人工智能
背景情况:
- 基础模型通常在大型数据集上进行预训练,然后为特定任务进行微调.
- 预培训数据集可能包含标签噪声,可能会损害模型的概括性,并引入风险.
- 预训练噪声对下游任务执行的影响仍然不充分理解.
研究的目的:
- 在预训练数据集中全面分析噪声的性质.
- 调查训练前噪声对域内和域外概括的影响.
- 提出和验证一种在基础模型中减轻噪声影响的方法.
主要方法:
- 使用合成噪音数据集 (ImageNet-1K,YFCC15M,CC12M) 进行广泛的实验,用于监督和对比的预训练.
- 在各种数据集尺度,噪音类型,模型架构和预培训目标中分析噪音影响.
- 开发和评估NMTune,一种用于特征空间调整的新调方法.
主要成果:
- 轻微的预训练噪声可以有利于域内性能,但始终降低域外性能.
- 这些发现在不同的数据集尺度,噪音类型,模型架构和预培训目标中都具有稳定性.
- NMTune有效地减轻噪声影响,并在参数效率和黑盒调场景中增强概括性.
结论:
- 预训练噪声从根本上改变了特征空间表示,影响了模型的概括.
- 拟议的NMTune方法提供了一个实际的解决方案,用于提高在噪音数据上训练的基础模型的稳定性.
- 这项研究确立了"噪音模型转移学习"作为未来研究的关键领域.
更多相关视频
08:05Measuring Statistical Learning Across Modalities and Domains in School-Aged Children Via an Online Platform and Neuroimaging Techniques
Published on: June 30, 2020
7.4K
09:22A Method for Remotely Silencing Neural Activity in Rodents During Discrete Phases of Learning
Published on: June 22, 2015
14.6K
相关概念视频
01:12Observational Learning
111
Albert Bandura's observational learning, also known as imitation or modeling, occurs when a person observes and imitates another's behavior. It is a quicker process than operant conditioning. A well-known example is the Bobo doll study, where children who saw an adult acting aggressively towards the doll were more likely to act aggressively when left alone, compared to those who observed a nonaggressive adult. Many psychologists view observational learning as a form of latent learning...
111
01:14Avoidance Learning and Learned Helplessness
1.7K
Avoidance learning and learned helplessness are critical concepts in understanding behavioral responses to negative stimuli.
Avoidance learning occurs when an organism learns that a specific behavior can prevent an unpleasant outcome. For example, a student who receives a bad grade may start studying harder to avoid future poor grades. This behavior persists even when the negative outcome is no longer present. Avoidance learning is powerful because it maintains behavior in the absence of the...
Avoidance learning occurs when an organism learns that a specific behavior can prevent an unpleasant outcome. For example, a student who receives a bad grade may start studying harder to avoid future poor grades. This behavior persists even when the negative outcome is no longer present. Avoidance learning is powerful because it maintains behavior in the absence of the...
1.7K
01:18Introduction to Learning
318
Learning is the process of acquiring knowledge or skills through practice or experience, leading to long-lasting behavioral changes. This acquisition occurs through interaction with the environment and requires practice or experience. For instance, mastering a skill such as surfing requires considerable practice and experience, highlighting the essential role of repeated interactions with the environment in learning.
In contrast to learned behaviors, unlearned behaviors such as crying, sexual...
In contrast to learned behaviors, unlearned behaviors such as crying, sexual...
318
01:24Generalization, Discrimination, and Extinction
375
Generalization, discrimination, and extinction are key concepts in operant conditioning that influence how behaviors are learned and maintained.
Generalization occurs when a behavior reinforced in one context is performed in similar situations. For instance, a student who studies diligently for calculus and receives excellent grades might apply the same study habits to psychology and history, expecting similar results. Generalization shows how learning in one setting can influence behavior in...
Generalization occurs when a behavior reinforced in one context is performed in similar situations. For instance, a student who studies diligently for calculus and receives excellent grades might apply the same study habits to psychology and history, expecting similar results. Generalization shows how learning in one setting can influence behavior in...
375
02:30Naturalistic Observations
15.3K
If you want to understand how behavior occurs, one of the best ways to gain information is to simply observe the behavior in its natural context. However, people might change their behavior in unexpected ways if they know they are being observed. How do researchers obtain accurate information when people tend to hide their natural behavior? As an example, imagine that your professor asks everyone in your class to raise their hand if they always wash their hands after using the restroom. Chances...
15.3K
01:24Effects of feedback
496
Feedback in control systems plays a critical role in shaping various operational parameters, extending beyond simple error reduction to influence stability, bandwidth, gain, impedance, and sensitivity. Understanding these effects requires examining a basic feedback system characterized by defined input, output, error, and feedback signals.
Feedback significantly modifies the gain of a control system. The gain of a system without feedback is altered by a factor of one plus GH, where G represents...
Feedback significantly modifies the gain of a control system. The gain of a system without feedback is altered by a factor of one plus GH, where G represents...
496