Toward robust and scalable deep spiking reinforcement learning.

Area of Science:

• Artificial Intelligence
• Computational Neuroscience

Background:

• Deep reinforcement learning (DRL) integrates reinforcement learning with deep neural networks (DNNs).
• Spiking neural networks (SNNs) offer a biologically plausible and energy-efficient alternative to DNNs.
• Surrogate gradient methods enable training SNNs using backpropagation through time (BPTT).

Purpose of the Study:

• Investigate the application of SNNs as function approximators in DRL algorithms.
• Evaluate SNN performance in discrete and continuous action space environments using DQN and TD3.

Main Methods:

• Applied SNNs to Deep Q-Network (DQN) and Twin-Delayed Deep Deterministic Policy Gradient (TD3) algorithms.
• Explored hyperparameter sensitivity, including membrane parameters like decay factors and thresholds.
• Investigated methods to improve SNN training stability, such as increasing simulation time and input encoding strategies.

Main Results:

• SNNs demonstrated sensitivity to hyperparameters like current and voltage decay factors, necessitating extensive tuning.
• Increased simulation time and a two-neuron encoding approach reduced sensitivity to membrane parameters.
• Randomizing membrane parameters, rather than using uniform values, enhanced training stability.

Conclusions:

• SNNs can be effectively utilized with state-of-the-art DRL algorithms for complex continuous control problems.
• While training complexity increases, SNNs offer potential for direct execution on energy-efficient neuromorphic processors.