Fully Spiking Actor Network With Intralayer Connections for Reinforcement Learning

Area of Science:

• Neuromorphic computing
• Artificial Intelligence
• Computational Neuroscience

Background:

• Spiking neural networks (SNNs) combined with deep reinforcement learning (DRL) offer energy-efficient AI solutions, particularly for realistic control tasks.
• Existing spike-based reinforcement learning methods often use firing rates, requiring floating-point operations that hinder direct deployment on neuromorphic hardware.
• Multidimensional deterministic policies are crucial for many real-world control scenarios.

Purpose of the Study:

• To develop a fully spiking actor network (SAN) that avoids floating-point matrix operations for seamless integration with neuromorphic hardware.
• To represent continuous action spaces using the membrane voltage of nonspiking neurons, inspired by insect neural mechanisms.
• To enhance the representation capacity of the output layer through intralayer connections in neuron populations.

Main Methods:

• Introduced nonspiking interneuron-inspired mechanisms where membrane voltage directly encodes action values.
• Utilized population neurons to decode different action dimensions, with neurons within each population connected in both time and space domains.
• Implemented intralayer connections within output populations to boost representational power, creating the intralayer connection SAN (ILC-SAN).

Main Results:

• The proposed ILC-SAN achieved state-of-the-art performance on continuous control tasks from OpenAI gym.
• Demonstrated superior performance compared to existing spike-based reinforcement learning methods.
• Estimated theoretical energy consumption for neuromorphic chip deployment, highlighting significant energy efficiency gains.

Conclusions:

• The ILC-SAN successfully enables fully spiking deep reinforcement learning for continuous control without floating-point operations.
• The proposed architecture is highly suitable for deployment on energy-efficient neuromorphic hardware.
• This work advances the field of energy-efficient AI and neuromorphic computing for complex control tasks.