在尖端神经网络中学习长序列.
Matei-Ioan Stan1, Oliver Rhodes2
1Department of Computer Science, The University of Manchester, Manchester, UK. matei.stan@manchester.ac.uk.
Scientific reports
|September 20, 2024
概括
国家空间模型 (SSM) 与尖端神经网络 (SNN) 结合,显示出对节能远程序列建模的前景. 这种方法在关键基准上优于变压器和当前SNN,为在神经形态硬件上高效的大型语言模型铺平了道路.
科学领域:
- 人工智能的人工智能
- 计算神经科学是一种神经科学.
- 机器学习 机器学习
背景情况:
- 尖端神经网络 (SNN) 提供节能计算,但由于RNN的限制和培训挑战,在顺序任务中落后于变压器.
- 国家空间模型 (SSM) 已经成为变压器的高效替代方案,用于序列建模.
研究的目的:
- 研究最先进的SSM与SNN的集成,用于长距离序列建模.
- 评估基于SSM的SNNs与变压器和现有SNNs的性能.
主要方法:
- 系统地调查SSM-SNN交叉点的长距离序列建模.
- 引入了一种新的特征混合层,以提高SNN的准确性.
- 与已建立的远程序列建模任务和顺序图像分类进行基准测试.
主要成果:
- 基于SSM的SNN在长距离序列建模基准中的所有任务中都超过了变压器模型.
- 基于SSM的SNNs在序列图像分类中具有较少参数的最先进的SNNs相比,实现了更高的性能.
- 一个新的特征混合层提高了SNN的准确性,质疑了关于二进制激活的先前假设.
结论:
- 基于SSM的SNN代表了节能远程序列建模的重大进步.
- 这项研究可以在神经形态硬件上部署强大的SSM架构,如大型语言模型.
- 这些发现为高效和脑启发的人工智能开辟了新的途径.
相关概念视频
Long-term Potentiation
54.9K
Long-term potentiation, or LTP, is one of the ways by which synaptic plasticity—changes in the strength of chemical synapses—can occur in the brain. LTP is the process of synaptic strengthening that occurs over time between pre- and postsynaptic neuronal connections. The synaptic strengthening of LTP works in opposition to the synaptic weakening of long-term depression (LTD) and together are the main mechanisms that underlie learning and memory.
54.9K
Neural Circuits
1.1K
Neural circuits and neuronal pools are two of the main structures found in the nervous system. Neural circuits are networks of neurons that work together to carry out a specific task or process. They consist of interconnected neurons and glial cells, which provide structural and metabolic support.
Neuronal pools are collections of nerve cells with similar functions and interact through chemical and electrical signals. These pools include both interneurons (the central neural circuit nodes that...
Neuronal pools are collections of nerve cells with similar functions and interact through chemical and electrical signals. These pools include both interneurons (the central neural circuit nodes that...
1.1K
The Role of Ion Channels in Neuronal Computation
3.2K
A postsynaptic neuron usually receives numerous impulses from several other presynaptic neurons. The axon hillock of the postsynaptic neuron integrates all these signals and determines the likelihood of firing an action potential.
Sometimes a single EPSP is strong enough to induce an action potential in the postsynaptic neuron. However, multiple presynaptic inputs must often create EPSPs around the same time for the postsynaptic neuron to be sufficiently depolarized to fire an action potential....
Sometimes a single EPSP is strong enough to induce an action potential in the postsynaptic neuron. However, multiple presynaptic inputs must often create EPSPs around the same time for the postsynaptic neuron to be sufficiently depolarized to fire an action potential....
3.2K


