Striatal Gradient in Value-Decay Explains Regional Differences in Dopamine Patterns and Reinforcement Learning...

Area of Science:

• Computational Neuroscience
• Reinforcement Learning Theory
• Neurobiology of Dopamine

Background:

• Dopamine's role in reward-prediction-error (RPE) is debated, with observed heterogeneous patterns across brain regions and conditions.
• Existing reinforcement learning (RL) theories struggle to reconcile these diverse dopamine signal patterns.
• The precise relationship between regional dopamine heterogeneity and various RL algorithms remains elusive.

Purpose of the Study:

• To demonstrate how incorporating value decay into RL models can coherently explain heterogeneous dopamine signals.
• To link value decay to specific RL algorithms, including predictive state representation, hierarchical RL, and distributional RL.
• To propose that a medial-to-lateral gradient of value or synaptic decay within the striatum tunes regional RL computations.

Main Methods:

• Developed computational models incorporating value decay, representing synaptic plasticity decay.
• Analyzed how value decay influences RPE signals under different state representations.
• Constructed hierarchical and distributional RL models with and without value decay to simulate striatal circuit activity.

Main Results:

• Value decay explains ramping RPE under specific state representations and accounts for fading dopamine ramping and cue-type/interval-dependent patterns.
• A coupled hierarchical RL model with and without value decay successfully replicated distinct striatal-dopamine circuit activity patterns.
• Distributional RL models with and without value decay elucidated how regional dopamine patterns relate to distributional coding strength.

Conclusions:

• Value decay, implemented via synaptic plasticity, provides a unified framework for understanding diverse dopamine RPE signals in RL.
• Within-striatum gradients in value or synaptic decay can tune regional RL computations by modulating dopamine/RPE signals.
• This framework reconciles regional dopamine heterogeneity with different RL algorithms, offering insights into flexible learning and habit formation.