Spiking Variational Policy Gradient for Brain Inspired Reinforcement Learning.

Area of Science:

• Computational Neuroscience
• Artificial Intelligence
• Machine Learning

Background:

• Reinforcement learning (RL) increasingly uses brain-inspired models for neuromorphic hardware.
• Reward-modulated spike-timing-dependent plasticity (R-STDP) is biologically plausible and energy-efficient but faces a local-global learning rule gap.
• This limitation hinders R-STDP's performance and broad applicability in complex tasks.

Purpose of the Study:

• To develop a novel R-STDP learning method that addresses the performance limitations caused by the local-global learning rule discrepancy.
• To enhance the capabilities of spiking neural networks in reinforcement learning by bridging local plasticity rules with global objectives.
• To improve the robustness and applicability of brain-inspired learning algorithms in artificial intelligence.

Main Methods:

• Designed a recurrent winner-take-all network architecture.
• Proposed the Spiking Variational Policy Gradient (SVPG), a new R-STDP learning method.
• Theoretically derived SVPG from global policy gradients, utilizing mean-field inference for policy function and a last-step approximation for policy optimization.

Main Results:

• SVPG successfully solved challenging tasks, including ViZDoom vision-based navigation and realistic robot control.
• Demonstrated superior inherent robustness of SVPG compared to existing methods against input variations, network parameters, and environmental perturbations.
• Effectively bridged the gap between local R-STDP learning rules and global learning objectives.

Conclusions:

• SVPG represents a significant advancement in biologically plausible reinforcement learning, overcoming key limitations of traditional R-STDP.
• The proposed method shows strong potential for applications in neuromorphic computing and advanced robotics.
• SVPG offers a robust and effective framework for brain-inspired artificial intelligence.