A spiking neural network model of an actor-critic learning agent

Area of Science:

• Computational neuroscience
• Artificial intelligence
• Behavioral adaptation

Background:

• Behavioral adaptation is essential for survival, with reinforcement learning offering effective strategies.
• The neural compatibility of temporal-difference learning algorithms remains an open question.

Purpose of the Study:

• To present a spiking neural network model implementing actor-critic temporal-difference learning.
• To investigate the neural plausibility of reinforcement learning algorithms.

Main Methods:

• Developed a spiking neural network model combining local plasticity rules and a global reward signal.
• Utilized a gridworld task with sparse rewards to test the network's learning capabilities.
• Derived a quantitative mapping between network parameters and algorithmic variables.

Main Results:

• The model successfully solved a nontrivial gridworld task with sparse rewards.
• Demonstrated that the spiking neural network learns at a comparable speed to discrete-time algorithms.
• Achieved equivalent equilibrium performance compared to standard reinforcement learning formulations.

Conclusions:

• The proposed spiking neural network model provides a biologically plausible implementation of actor-critic temporal-difference learning.
• The findings suggest a strong compatibility between temporal-difference learning and neural computation.
• The model offers a framework for understanding adaptive behavior in biological and artificial systems.