关于无监督预训练的概括能力
Yuyang Deng1, Junyuan Hong2, Jiayu Zhou2
1Penn State University.
概括
这项研究引入了一个新的理论,解释了无监督的预训练如何改善模型概括. 它揭示了知识可转移性的关键因素,提高了微调模型在下游任务上的性能.
科学领域:
- 机器学习 机器学习
- 人工智能的人工智能
- 深度学习 (Deep Learning) 是一种深度学习.
背景情况:
- 无监督的预训练之后的微调增强了模型的概括性.
- 在理解前培训表示如何影响微调模型概括方面存在理论上的差距,特别是考虑到分布和任务异质性.
研究的目的:
- 开发一种新的理论框架来分析未经监督的预培训知识的可转移性.
- 阐明影响下游任务微调模型概括的关键因素.
主要方法:
- 开发了一个新的理论框架来分析无监督的预训练和微调.
- 应用框架来分析Context Encoder和Masked Autoencoder预培训方法的概括界限.
- 研究深度神经网络和深度变压器进行预训练,随后进行二进制分类微调.
主要成果:
- 确定了影响知识可转移性的关键因素,从预培训到微调.
- 为特定的预培训场景提供了概括界限的理论分析.
- 提出了一种新的规范化方法,以改善微调的模型通用化.
结论:
- 该研究提供了对无监督预训练和微调范式的更深入的理解.
- 这些发现可以指导开发更有效的预训练算法,以提高概括性.
相关概念视频
Generalization, Discrimination, and Extinction
458
Generalization, discrimination, and extinction are key concepts in operant conditioning that influence how behaviors are learned and maintained.
Generalization occurs when a behavior reinforced in one context is performed in similar situations. For instance, a student who studies diligently for calculus and receives excellent grades might apply the same study habits to psychology and history, expecting similar results. Generalization shows how learning in one setting can influence behavior in...
Generalization occurs when a behavior reinforced in one context is performed in similar situations. For instance, a student who studies diligently for calculus and receives excellent grades might apply the same study habits to psychology and history, expecting similar results. Generalization shows how learning in one setting can influence behavior in...
458
Case Studies
11.6K
There are many research methods available to psychologists in their efforts to understand, describe, and explain behavior and the cognitive and biological processes that underlie it.
11.6K
Real-World Application of Classical Conditioning
531
Classical conditioning not only includes the initial pairing of stimuli but also extends to more complex forms, such as higher-order conditioning. Higher-order conditioning involves creating associations beyond the primary conditioned stimulus, resulting in a chain of conditioned responses.
Higher-order, or second-order, conditioning occurs when a neutral stimulus becomes associated with an already established conditioned stimulus through repeated pairings. For instance, if a dog has been...
Higher-order, or second-order, conditioning occurs when a neutral stimulus becomes associated with an already established conditioned stimulus through repeated pairings. For instance, if a dog has been...
531
The Representativeness Heuristic
15.8K
The representative heuristic describes a biased way of thinking, in which you unintentionally stereotype someone or something. For example, you may assume that your professors spend their free time reading books and engaging in intellectual conversation, because the idea of them spending their time playing volleyball or visiting an amusement park does not fit in with your stereotypes of professors.
15.8K
Inductive Reasoning
60.1K
Inductive reasoning is a form of logical thinking that uses related observations to arrive at a general conclusion. It is uncertain and operates in degrees to which the conclusions are credible. As such, inductive arguments can be weak or strong, rather than valid or invalid, and conclusions can be used to formulate testable, falsifiable hypotheses.
Inductive reasoning is common in descriptive science. A life scientist makes observations and records them. This data can be qualitative or...
Inductive reasoning is common in descriptive science. A life scientist makes observations and records them. This data can be qualitative or...
60.1K
Stereotype Content Model
14.0K
The Stereotype Content Model (SCM) was first proposed by Susan Fiske and her colleagues (Fiske, Cuddy, Glick & Xu, 2002; see also Fiske, 2012 and Fiske, 2017). The SCM specifies that when someone encounters a new group, they will stereotype them based on two metrics: warmth—or that group’s perceived intent, and how likely they are to provide help or inflict harm—and competence—or their ability to carry out that objective. Depending on the warmth-competence...
14.0K


