通过线性注意力在语境内学习的非对称理论
Yue M Lu1, Mary Letey1, Jacob A Zavatone-Veth1,2,3,4
1The John A. Paulson School of Engineering and Applied Sciences, Harvard University, Cambridge, MA 02138.
概括
变压器在没有事先培训的情况下表现出上下文学习 (ICL). 这项研究精确地在线性注意力中模拟了ICL,揭示了双下降曲线和影响概括与记忆的阶段过渡.
科学领域:
- 人工智能的人工智能
- 机器学习 机器学习
- 深度学习 (Deep Learning) 是一种深度学习.
背景情况:
- 变压器展示了上下文学习 (ICL),这是执行任务的关键能力,没有明确的培训.
- 关于样本复杂性,预训练任务多样性以及对于有效的ICL至关重要的上下文长度,仍然存在未解决的问题.
研究的目的:
- 准确地回答有关ICL要求的问题,使用一个完全可解决的模型.
- 分析预训练任务多样性和样本复杂性的对变压器概括的影响.
主要方法:
- 开发了一个完全可解决的ICL模型,用于使用线性注意力的线性回归任务.
- 在特定的缩放模式下,为学习曲线推导出了尖的非对称性 (无限的标记维度,比例的上下文长度和任务多样性,二进制的预训练示例).
主要成果:
- 随着预训练示例的增加,演示了双下降的学习曲线.
- 确定了低和高任务多样性制度之间的阶段过渡,将记忆与真正的ICL和概括区分开来.
结论:
- 对ICL机制的理论见解得到了推导,并经验验证.
- 对于变形器来说,高任务多样性是必要的,以实现真正的ICL,并将其推广到预先训练的任务之外.
更多相关视频
08:05Measuring Statistical Learning Across Modalities and Domains in School-Aged Children Via an Online Platform and Neuroimaging Techniques
Published on: June 30, 2020
7.7K
07:36Eye Tracking During Visually Situated Language Comprehension: Flexibility and Limitations in Uncovering Visual Context Effects
Published on: November 30, 2018
15.8K
相关概念视频
Cognitive Learning
513
Cognitive learning is based on purposive behavior, incidental learning, and insight learning.
E. C. Tolman's theory of purposive behavior emphasizes that much behavior is goal-directed. He argued that to understand behavior, we must look at the entire sequence of actions leading to a goal. For instance, high school students study hard, not just due to past reinforcement but also to achieve the goal of getting into a good college.
Tolman introduced the idea that behavior is influenced by...
E. C. Tolman's theory of purposive behavior emphasizes that much behavior is goal-directed. He argued that to understand behavior, we must look at the entire sequence of actions leading to a goal. For instance, high school students study hard, not just due to past reinforcement but also to achieve the goal of getting into a good college.
Tolman introduced the idea that behavior is influenced by...
513
Purposive Learning
203
E. C. Tolman emphasized the purposiveness of behavior — the idea that much of our behavior is goal-directed. For instance, employees who aim for a promotion work diligently to meet their targets. Tolman argued that when classical conditioning and operant conditioning occur, the organism acquires certain expectations. In classical conditioning, a child might fear a dog because they expect it to bite. In operant conditioning, a person might consistently work overtime because they expect a...
203
Associative Learning
566
Associative learning is a fundamental concept in behavioral psychology, wherein a connection is established between two stimuli or events, leading to a learned response. This process is critical in understanding how behaviors are acquired and modified. Conditioning, the mechanism through which associations are formed, can be divided into two main types: classical conditioning and operant conditioning, each elucidating different aspects of associative learning.
Classical conditioning, also known...
Classical conditioning, also known...
566
Observational Learning
305
Albert Bandura's observational learning, also known as imitation or modeling, occurs when a person observes and imitates another's behavior. It is a quicker process than operant conditioning. A well-known example is the Bobo doll study, where children who saw an adult acting aggressively towards the doll were more likely to act aggressively when left alone, compared to those who observed a nonaggressive adult. Many psychologists view observational learning as a form of latent learning...
305
Linear time-invariant Systems
398
A system is linear if it displays the characteristics of homogeneity and additivity, together termed the superposition property. This principle is fundamental in all linear systems. Linear time-invariant (LTI) systems include systems with linear elements and constant parameters.
The input-output behavior of an LTI system can be fully defined by its response to an impulsive excitation at its input. Once this impulse response is known, the system's reaction to any other input can be...
The input-output behavior of an LTI system can be fully defined by its response to an impulsive excitation at its input. Once this impulse response is known, the system's reaction to any other input can be...
398
Higher Mental Functions of Brain: Learning and Memory
963
Memory is one of the most vital higher mental functions of the brain. Memory is closely related to learning because it enables us to retain information and experiences from our past to use them in our present life. It also helps us to remember facts, events, and skills, such as riding a bike or swimming. There are two types of memory — declarative memory, which involves memorizing facts or events, and procedural memory, which enables us to remember how to do something like writing or...
963
