Dopamine encoding of novelty facilitates efficient uncertainty-driven exploration.

Area of Science:

• Neuroscience
• Computational Neuroscience
• Reinforcement Learning

Background:

• Animals must explore unfamiliar environments to learn action-reward associations and optimize decision-making.
• Effective exploration requires assessing and utilizing uncertainties in action-reward knowledge.

Purpose of the Study:

• To propose a novel computational model of the basal ganglia for uncertainty-driven exploration.
• To investigate the role of striatal pathways and dopaminergic neurons in estimating reward uncertainty and novelty signals.

Main Methods:

• Developed a computational model of basal ganglia function, integrating direct and indirect striatal pathways for reward mean and variance estimation.
• Utilized electrophysiological data to validate the basal ganglia model.
• Fitted exploration strategies derived from the neural model to behavioral data.
• Compared model-inspired exploration strategies against classic algorithms like Upper Confidence Bound (UCB) in simulations.

Main Results:

• The proposed basal ganglia model effectively estimates reward distribution means and variances.
• Electrophysiological data supported the model's representation of basal ganglia function.
• Strategies inspired by the basal ganglia model demonstrated superior performance in simulations compared to UCB.
• Model-fitting to behavioral data yielded results comparable to idealized normative models.

Conclusions:

• Transient dopamine signals encoding novelty in the basal ganglia contribute to uncertainty representation.
• This uncertainty representation efficiently drives exploration in reinforcement learning.
• The model provides a biologically plausible mechanism for optimal decision-making under uncertainty.