相关实验视频

Updated: Sep 16, 2025

A Simple Stimulatory Device for Evoking Point-like Tactile Stimuli: A Searchlight for LFP to Spike Transitions

A Simple Stimulatory Device for Evoking Point-like Tactile Stimuli: A Searchlight for LFP to Spike Transitions

Published on: March 25, 2014

结合聚合注意力和变压器架构,实现尖端神经网络的准确和高效性能

Hangming Zhang¹, Alexander Sboev², Roman Rybka²

¹College of Intelligence and Computing, Tianjin University, Tianjin, 300354, China.

Neural networks : the official journal of the International Neural Network Society

|July 9, 2025

概括

本研究介绍了SAFormer,这是一种新的尖端神经网络 (SNN) 和变压器模型,可以在低能耗的情况下实现高性能. 澳大利亚前南非人

科学领域:

人工智能的人工智能
深度学习 (Deep Learning) 是一种深度学习.
神经形态计算是一种神经形态计算.

背景情况:

尖端神经网络 (SNN) 通过模仿生物神经元提供低功耗计算.
由于自我注意力机制,变压器模型在各个领域的性能非常出色.
在低功耗场景中,集成SNN和变压器是具有挑战性的,因为传统的注意力机制在低功耗场景中的计算需求.

研究的目的:

提出一种新的模型架构,SAFormer,它将SNN的低功耗优势与变压器的高性能相结合.
解决SNNs和变压器集成的挑战,以实现高效的低功耗,高性能计算.
在不影响精度的情况下,降低基于变压器的模型的能源消耗.

主要方法:

引入了尖峰聚合自我注意力 (SASA) 机制,它简化了仅使用尖峰矩阵 (查询和键) 的注意力重量计算.
整合了一个深度卷积模块 (DWC) 来增强特征提取能力.
在多个基准数据集上评估SAFormer模型,包括CIFAR-10,CIFAR-100,Tiny-ImageNet,DVS128-Gesture和CIFAR10-DVS.

主要成果:

与最先进的SNNs相比,SAFormer表现出更高的性能.
该模型实现了能源消耗的显著降低.
由于DWC模块的集成,观察到更高的准确性.

结论:

SAFormer有效地将SNN的低功耗特性与变压器的高性能优势相结合.
通过简化注意力计算,SASA机制显著降低了能源消耗.
在低功耗环境中,SAFormer为开发高效,高性能的人工智能模型提供了一个有前途的方法.

关键词:

能源效率能源效率是指能源的使用效率.尖峰集成的自我注意力.尖端神经网络的神经网络.变压器变压器变压器

更多相关视频

Swin-PSAxialNet: An Efficient Multi-Organ Segmentation Technique

Swin-PSAxialNet: An Efficient Multi-Organ Segmentation Technique

Published on: July 5, 2024

A Swin Transformer-Based Model for Thyroid Nodule Detection in Ultrasound Images

A Swin Transformer-Based Model for Thyroid Nodule Detection in Ultrasound Images

Published on: April 21, 2023

相关概念视频

Neural Circuits

Neural Circuits

Neural circuits and neuronal pools are two of the main structures found in the nervous system. Neural circuits are networks of neurons that work together to carry out a specific task or process. They consist of interconnected neurons and glial cells, which provide structural and metabolic support.
Neuronal pools are collections of nerve cells with similar functions and interact through chemical and electrical signals. These pools include both interneurons (the central neural circuit nodes that...

Graded Potential

Graded Potential

Graded potentials are localized fluctuations in the cell membrane's electrical charge, commonly found in the dendrites of neurons. The magnitude of these potential changes depends on the strength of the initiating stimulus. In a membrane at its resting potential, a graded potential signifies a voltage shift either above -70 mV or below -70 mV.
Graded potentials fall into two categories: depolarizing and hyperpolarizing. Depolarizing graded potentials typically occur when sodium (Na+) or...

Integration of Synaptic Events

Integration of Synaptic Events

Synaptic integration mainly includes the summation of graded potentials. Graded potentials, regardless of their type, cause subtle alterations in membrane voltage, resulting in either depolarization or hyperpolarization. These incremental changes, when combined or summed, can propel the neuron toward its threshold. Consider, for example, a membrane experiencing a +15 mV shift, causing it to depolarize from -70 mV to -55 mV. In this scenario, graded potentials govern the membrane's ability to...