深度线性网络的确切学习动态,具有先前知识
Clémentine C J Dominé1, Lukas Braun2, James E Fitzgerald3
1Gatsby Computational Neuroscience Unit, University College London, London, United Kingdom.
概括
深度神经网络的学习依赖于初始权重. 本研究为深度线性网络中的学习动态提供了准确的解决方案,揭示了初始化如何影响学习速度和融合.
科学领域:
- 机器学习 机器学习
- 深度学习理论 深度学习理论
- 计算神经科学是一种神经科学.
背景情况:
- 深度神经网络 (DNN) 的学习效率受到初始网络权重的严重影响.
- 对初始权重的先前知识如何塑造学习动态的理论理解仍然有限.
研究的目的:
- 在具有丰富的先验知识的深度线性网络中获得学习动态的确切解决方案.
- 概括现有的理论框架来分析DNN中的学习.
主要方法:
- 福木子矩阵里卡蒂解决方案的概括.
- 为不断演变的网络功能,表示相似性和神经触角内核衍生明确表达式.
- 对广泛类型的初始化和任务进行分析.
主要成果:
- 确定了独立于任务的初始化,可以加速学习动态,从缓慢到快速的指数轨迹.
- 证明了对全球最佳的趋同,保留了独立于初始内部表示的表示相似性.
- 描述了网络权重与任务结构的动态对齐.
结论:
- 开发了一个数学工具包,以了解先前知识对深度学习动态的影响.
- 为以前的学习模式提供了严格的理由,并强调了对持续和反转学习的影响.
- 展示了特定的初始化如何将学习轨迹与初始表示结构脱.
相关概念视频
Associative Learning
353
Associative learning is a fundamental concept in behavioral psychology, wherein a connection is established between two stimuli or events, leading to a learned response. This process is critical in understanding how behaviors are acquired and modified. Conditioning, the mechanism through which associations are formed, can be divided into two main types: classical conditioning and operant conditioning, each elucidating different aspects of associative learning.
Classical conditioning, also known...
Classical conditioning, also known...
353
Purposive Learning
119
E. C. Tolman emphasized the purposiveness of behavior — the idea that much of our behavior is goal-directed. For instance, employees who aim for a promotion work diligently to meet their targets. Tolman argued that when classical conditioning and operant conditioning occur, the organism acquires certain expectations. In classical conditioning, a child might fear a dog because they expect it to bite. In operant conditioning, a person might consistently work overtime because they expect a...
119
Cognitive Learning
239
Cognitive learning is based on purposive behavior, incidental learning, and insight learning.
E. C. Tolman's theory of purposive behavior emphasizes that much behavior is goal-directed. He argued that to understand behavior, we must look at the entire sequence of actions leading to a goal. For instance, high school students study hard, not just due to past reinforcement but also to achieve the goal of getting into a good college.
Tolman introduced the idea that behavior is influenced by...
E. C. Tolman's theory of purposive behavior emphasizes that much behavior is goal-directed. He argued that to understand behavior, we must look at the entire sequence of actions leading to a goal. For instance, high school students study hard, not just due to past reinforcement but also to achieve the goal of getting into a good college.
Tolman introduced the idea that behavior is influenced by...
239
Linear time-invariant Systems
254
A system is linear if it displays the characteristics of homogeneity and additivity, together termed the superposition property. This principle is fundamental in all linear systems. Linear time-invariant (LTI) systems include systems with linear elements and constant parameters.
The input-output behavior of an LTI system can be fully defined by its response to an impulsive excitation at its input. Once this impulse response is known, the system's reaction to any other input can be...
The input-output behavior of an LTI system can be fully defined by its response to an impulsive excitation at its input. Once this impulse response is known, the system's reaction to any other input can be...
254
Observational Learning
170
Albert Bandura's observational learning, also known as imitation or modeling, occurs when a person observes and imitates another's behavior. It is a quicker process than operant conditioning. A well-known example is the Bobo doll study, where children who saw an adult acting aggressively towards the doll were more likely to act aggressively when left alone, compared to those who observed a nonaggressive adult. Many psychologists view observational learning as a form of latent learning...
170
Classification of Systems-I
184
Linearity is a system property characterized by a direct input-output relationship, combining homogeneity and additivity.
Homogeneity dictates that if an input x(t) is multiplied by a constant c, the output y(t) is multiplied by the same constant. Mathematically, this is expressed as:
Homogeneity dictates that if an input x(t) is multiplied by a constant c, the output y(t) is multiplied by the same constant. Mathematically, this is expressed as:
184


