Multi-timescale reinforcement learning in the brain

Area of Science:

• Neuroscience
• Computational Neuroscience
• Machine Learning

Background:

• Adaptive behavior in complex environments requires maximizing rewards.
• Reinforcement learning (RL) models this adaptive behavior and characterizes dopaminergic neuron activity.
• Classical RL uses a single discount factor for future rewards.

Purpose of the Study:

• To investigate the role of multiple timescales in biological reinforcement learning.
• To explore the computational benefits of agents learning at various timescales.
• To characterize the temporal discounting properties of dopaminergic neurons.

Main Methods:

• Developed reinforcement learning agents with multiple learning timescales.
• Recorded dopaminergic neuron activity in mice during two behavioral tasks.
• Modeled reward prediction error and temporal discounting in neural responses.

Main Results:

• Reinforcement agents with multiple timescales demonstrated enhanced computational benefits.
• Dopaminergic neurons in mice showed a diversity of discount time constants.
• A model explained neural heterogeneity in temporal discounting, including dopamine ramps.
• Individual neuron discount factors were consistent across tasks, indicating cell-specific properties.

Conclusions:

• Multiple timescales are crucial for efficient biological reinforcement learning.
• Dopaminergic neuron heterogeneity in temporal discounting provides a mechanistic basis for non-exponential reward valuation.
• Findings offer a new framework for understanding dopaminergic function and designing advanced RL algorithms.