Learning the payoffs and costs of actions

Area of Science:

• Neuroscience
• Computational Neuroscience
• Reinforcement Learning

Background:

• The basal ganglia, a group of sub-cortical nuclei, are crucial for action value learning.
• Two distinct pathways within the basal ganglia are linked to approach and avoidance behaviors.
• Dopamine projections from the midbrain differentially modulate these pathways.

Purpose of the Study:

• To investigate if basal ganglia pathways can represent learned estimates of action consequences (payoffs and costs).
• To model how dopamine activity influences the evaluation of action payoffs and costs.
• To explore the suitability of specific plasticity rules for extracting these consequences from prediction errors.

Main Methods:

• Development of a computational model inspired by the opponent actor learning framework.
• Simulation of plasticity rules for extracting payoffs and costs from temporally distinct prediction errors.
• Analysis of the model's ability to reproduce drug-induced changes in motivation.

Main Results:

• The model demonstrates that basal ganglia pathways can represent learned payoffs and costs.
• Dopamine activity level modulates the influence of payoffs and costs on action evaluation.
• Specific plasticity rules require differential prediction error effects and weak synaptic decay for successful learning.
• Simulations successfully reproduced drug-induced changes in willingness to work, consistent with haloperidol experiments.

Conclusions:

• Basal ganglia pathways may encode learned estimates of positive and negative action consequences.
• Dopamine plays a key role in regulating motivational states and action valuation.
• Established plasticity rules are effective for learning action values when outcomes are temporally separated.
• The model provides a framework for understanding dopamine's role in motivation and learning.