Dopaminergic Balance between Reward Maximization and Policy Complexity.

Area of Science:

• Neuroscience
• Computational Neuroscience
• Reinforcement Learning

Background:

• Dopamine's role in reinforcement learning is established, primarily for prediction errors.
• The computational goals of the basal ganglia's actor system remain less explored.

Purpose of the Study:

• To develop a top-down model of the basal ganglia emphasizing dopamine's function.
• To investigate how the basal ganglia balance policy complexity and reward gain.
• To propose an experience-modulated softmax policy as a framework for basal ganglia function.

Main Methods:

• Constructed a computational model of the basal ganglia network.
• Incorporated dopamine's role as both a reinforcement signal and a pseudo-temperature regulator.
• Analyzed the trade-off between minimizing policy complexity (cost) and maximizing future reward (gain).

Main Results:

• The model demonstrates that multi-dimensional optimization leads to an experience-modulated softmax policy.
• Action selection probability depends on estimated values, pseudo-temperature, and action frequency.
• Dopamine modulates striatal excitability, influencing the gain-cost trade-off.

Conclusions:

• The basal ganglia's computational goal is optimizing independent gain and cost functions, not just cumulative reward.
• This multi-dimensional optimization naturally yields a softmax-like behavioral policy.
• The proposed framework accounts for diverse behaviors and clinical states regulated by basal ganglia and dopamine systems.