Uncertainty-guided learning with scaled prediction errors in the basal ganglia

Area of Science:

• Neuroscience
• Computational Neuroscience
• Reinforcement Learning

Background:

• Accurate reward prediction in biological systems requires accounting for observation variability, especially noise.
• Current models lack a clear mechanism for tracking and utilizing observation noise magnitude in reward prediction updates.
• The brain's reward system's ability to adapt to noisy environments is not fully understood.

Purpose of the Study:

• To introduce a new computational model for reward prediction that incorporates observation noise.
• To investigate how the brain might track reward variability and use it to modulate learning.
• To propose a biological implementation of this model in the basal ganglia circuit.

Main Methods:

• Developed a novel reinforcement learning model with learning rules to track reward mean and standard deviation.
• Utilized prediction errors scaled by uncertainty as the primary feedback signal.
• Proposed a neural network architecture in the basal ganglia for biological plausibility.

Main Results:

• The new model demonstrated superior performance in value tracking tasks compared to conventional reinforcement learning models.
• The model's performance approached the theoretical optimum set by the Kalman filter.
• Simulated dopaminergic neurons encoded reward prediction errors scaled by reward standard deviation, consistent with experimental data.

Conclusions:

• The proposed model offers a computationally tractable and biologically plausible mechanism for adaptive reward learning under noisy conditions.
• The findings suggest that the basal ganglia may track reward uncertainty to modulate dopaminergic signals.
• This work has implications for understanding dopamine's role in learning and decision-making, particularly in variable environments.