Reinforcement learning using a continuous time actor-critic framework with spiking neurons.

Area of Science:

• Computational Neuroscience
• Reinforcement Learning
• Neurobiology

Background:

• Dopamine's role in reward and synaptic plasticity is known, but its precise computation in learning is unclear.
• Reinforcement learning (RL) theory offers a framework, but discrete models don't fit natural behaviors.
• Biologically plausible models struggle to compute reward prediction errors (RPEs) in spiking neurons.

Purpose of the Study:

• To bridge the gap between RL and neuroscience by developing a biologically plausible model for reward-modulated plasticity.
• To address limitations of discrete RL frameworks and the computation of RPEs in spiking neural networks.
• To propose a continuous-time actor-critic model for spiking neurons.

Main Methods:

• Extended continuous temporal difference (TD) learning to an actor-critic network with spiking neurons.
• Implemented continuous time, state, and action representations.
• Simulated the model on navigation and complex motor control tasks.

Main Results:

• The model successfully solved a Morris water-maze-like navigation task, matching animal performance.
• The architecture efficiently solved the acrobot and cartpole control problems.
• The derived learning rule aligns with experimental findings on dopamine-modulated plasticity.

Conclusions:

• The proposed model offers a plausible neural mechanism for computing reward prediction errors.
• This framework integrates continuous RL with spiking neural network dynamics.
• The findings support a biological basis for reward-modulated synaptic plasticity.