An opponent striatal circuit for distributional reinforcement learning.

Area of Science:

• Neuroscience
• Computational Neuroscience
• Machine Learning

Background:

• Machine learning advances utilize distributional reinforcement learning (RL) by considering entire reward distributions.
• The mammalian brain's mesolimbic dopamine system is implicated in RL, primarily through mean value updates in the striatum.
• Limited knowledge exists on how striatal neurons encode higher-order moments of reward distributions.

Purpose of the Study:

• Investigate neural encoding of reward distribution moments in the mammalian brain.
• Determine the role of the striatum and dopamine in distributional RL.
• Elucidate the contribution of different striatal neuron types to reward variance coding.

Main Methods:

• High-density Neuropixels recordings of striatal activity in mice during a classical conditioning task.
• Independent manipulation of reward mean, variance, and stimulus identity.
• Chronic dopamine input ablation, two-photon calcium imaging, and optogenetics.

Main Results:

• Striatal neurons exhibit abstract encoding of reward variance, challenging traditional RL models.
• Dopamine input ablation disrupted distributional representations but not mean value coding.
• D1 and D2 medium spiny neurons differentially encoded the right and left tails of the reward distribution, respectively.

Conclusions:

• The striatum and mesolimbic dopamine system support distributional RL through abstract variance encoding.
• Dopamine plays a crucial role in maintaining these distributional representations.
• A novel model of striatal function leverages D1/D2 neuron opponency for distributional RL benefits.