从现代霍普菲尔德网络中出现的博尔兹曼机器家族中的注意力
Toshihiro Ota1,2, Ryo Karakida3
1CyberAgent, Shibuya, Tokyo, 150-6121, Japan.
Neural computation
|July 11, 2023
概括
本研究介绍了注意力博尔茨曼机器 (AttnBM),这是一个基于现代霍普菲尔德网络的基于新能源的模型. AttnBM展示了可训练性,并揭示了与其他神经网络模型的连接.
科学领域:
- 人工智能的人工智能
- 机器学习 机器学习
- 计算神经科学是一种神经科学.
背景情况:
- 霍普菲尔德网络和博尔兹曼机器 (BM) 是基础的基于能量的神经网络模型.
- 最近的进展已经统一了一般的霍普菲尔德网络,包括注意力机制.
- 了解BM对应物的可训练性对于推进这些模型至关重要.
研究的目的:
- 为了探索博尔兹曼机器对应现代霍普菲尔德网络的特性.
- 介绍和分析一个新的注意力博尔兹曼机器 (AttnBM).
- 调查AttnBM的可训练性和理论联系.
主要方法:
- 从现代的霍普菲尔德网络能量函数中获得博尔兹曼机器对应物.
- 分析拟议的注意力博尔兹曼机器 (AttnBM) 的特性.
- 调查与Gaussian-Bernoulli等现有模型的连接限制了BMs,并否认了自动编码器.
主要成果:
- 引入了注意力博尔兹曼机器 (AttnBM),它是由现代霍普菲尔德网络中的注意力模块衍生而来的.
- 在特定情况下,AttnBM表现出可处理的概率函数和梯度,使训练更容易.
- 发现了AttnBM和单层模型之间的隐藏连接,包括denoising分数匹配.
- 从密集的关联记忆模型中衍生出的博尔兹曼机器被证明属于指数式家庭和.
结论:
- 注意力博尔兹曼机器 (AttnBM) 为基于能源的模型提供了一个有前途的新方向.
- 该研究强调了AttnBM.的可训练性优势和理论联系.
- 这项工作加深了对霍普菲尔德网络和博尔兹曼机器之间的关系的理解.
相关概念视频
01:19The Role of Ion Channels in Neuronal Computation
3.2K
A postsynaptic neuron usually receives numerous impulses from several other presynaptic neurons. The axon hillock of the postsynaptic neuron integrates all these signals and determines the likelihood of firing an action potential.
Sometimes a single EPSP is strong enough to induce an action potential in the postsynaptic neuron. However, multiple presynaptic inputs must often create EPSPs around the same time for the postsynaptic neuron to be sufficiently depolarized to fire an action potential....
Sometimes a single EPSP is strong enough to induce an action potential in the postsynaptic neuron. However, multiple presynaptic inputs must often create EPSPs around the same time for the postsynaptic neuron to be sufficiently depolarized to fire an action potential....
3.2K
01:23Atomic Nuclei: Nuclear Relaxation Processes
683
In the absence of an external magnetic field, nuclear spin states are degenerate and randomly oriented. When a magnetic field is applied, the spins begin to precess and orient themselves along (lower energy) or against (higher energy) the direction of the field. At equilibrium, a slight excess population of spins exists in the lower energy state. Because the direction of the magnetic field is fixed as the z-axis, the precessing magnetic moments are randomly oriented around the z-axis.
683
01:31Action Potential
8.0K
Neurons communicate by firing action potentials—the electrochemical signal that is propagated along the axon. The signal results in the release of neurotransmitters at axon terminals, thereby transmitting information to the nervous system. An action potential is a specific "all-or-none" change in membrane potential that results in a rapid spike in voltage.
Membrane potential in neurons
Neurons typically have a resting membrane potential of about -70 millivolts (mV). When they...
Membrane potential in neurons
Neurons typically have a resting membrane potential of about -70 millivolts (mV). When they...
8.0K
01:21Cognitive Learning
432
Cognitive learning is based on purposive behavior, incidental learning, and insight learning.
E. C. Tolman's theory of purposive behavior emphasizes that much behavior is goal-directed. He argued that to understand behavior, we must look at the entire sequence of actions leading to a goal. For instance, high school students study hard, not just due to past reinforcement but also to achieve the goal of getting into a good college.
Tolman introduced the idea that behavior is influenced by...
E. C. Tolman's theory of purposive behavior emphasizes that much behavior is goal-directed. He argued that to understand behavior, we must look at the entire sequence of actions leading to a goal. For instance, high school students study hard, not just due to past reinforcement but also to achieve the goal of getting into a good college.
Tolman introduced the idea that behavior is influenced by...
432
01:25Neural Circuits
1.3K
Neural circuits and neuronal pools are two of the main structures found in the nervous system. Neural circuits are networks of neurons that work together to carry out a specific task or process. They consist of interconnected neurons and glial cells, which provide structural and metabolic support.
Neuronal pools are collections of nerve cells with similar functions and interact through chemical and electrical signals. These pools include both interneurons (the central neural circuit nodes that...
Neuronal pools are collections of nerve cells with similar functions and interact through chemical and electrical signals. These pools include both interneurons (the central neural circuit nodes that...
1.3K
01:28Neuronal Communication
1.0K
Neurons, the fundamental units of the brain and nervous system, communicate through complex electrochemical signals that underpin all cognitive and bodily functions. This communication is primarily facilitated by a process involving the generation and propagation of an action potential along the axon of the neuron. When the internal electrical charge of a neuron surpasses a certain threshold, an action potential is triggered. This rapid change in voltage travels swiftly along the axon to the...
1.0K