Neuronal attention circuit (NAC) for representation learning

Area of Science:

• Computational Neuroscience
• Machine Learning
• Artificial Intelligence

Background:

• Attention mechanisms enhance Recurrent Neural Networks (RNNs) but their discrete nature poses challenges for continuous-time (CT) modeling.
• Biologically inspired neural circuits offer potential for novel computational approaches.

Purpose of the Study:

• To introduce the Neuronal Attention Circuit (NAC), a novel biologically inspired CT-attention mechanism.
• To address limitations of discrete attention in CT modeling and improve representation learning.

Main Methods:

• Reformulated attention logit computation as a solution to a linear first-order ODE using nonlinear gates from C. elegans Neuronal Circuit Policies (NCPs).
• Implemented sparse sensory gates for query-key projections and a sparse backbone network with two heads.
• Utilized an adaptable, sparse, subquadratic Top-K pairwise concatenation for efficient query-key interactions.

Main Results:

• NAC demonstrates competitive or superior accuracy compared to state-of-the-art CT baselines across diverse domains.
• Achieved efficient adaptive dynamics with improved efficiency and memory consumption.
• Provided theoretical guarantees for state stability and bounded approximation errors.

Conclusions:

• NAC offers a biologically plausible and computationally efficient CT-attention mechanism.
• The model shows strong performance in irregular time-series, long-range forecasting, autonomous driving, and industrial prognostics.
• NAC provides neuron-level interpretability, enhancing understanding of its decision-making processes.