Navigating complex decision spaces: Problems and paradigms in sequential choice.

Area of Science:

• Neuroscience
• Cognitive Science
• Computational Neuroscience

Background:

• Adaptive behavior requires learning from action consequences, which is challenging with delayed feedback.
• The temporal credit assignment problem addresses how to attribute outcomes to intermediate actions in a sequence.
• Reinforcement learning offers model-free and model-based approaches to solve this problem.

Purpose of the Study:

• To review connections between learning theories, control mechanisms, and reinforcement learning models.
• To examine temporal credit assignment problems through the lens of reinforcement learning.
• To identify neural substrates associated with model-free and model-based reinforcement learning.

Main Methods:

• Literature review connecting stimulus-response and cognitive learning theories.
• Examination of habitual versus goal-directed control.
• Analysis of model-free and model-based reinforcement learning strategies.
• Review of problems including second-order conditioning, latent learning, and hierarchical learning.
• Investigation of neural substrates for reinforcement learning.

Main Results:

• Model-free reinforcement learning is linked to dopamine and basal ganglia.
• Model-based reinforcement learning involves prefrontal cortex, medial temporal lobes, cerebellum, and basal ganglia.
• Reinforcement learning provides a framework for understanding animal and human behavior.
• Reinforcement learning models offer insights into neural reward valuation and action selection.

Conclusions:

• Reinforcement learning provides a robust framework for understanding temporal credit assignment in behavior and neural mechanisms.
• Distinct neural circuits support model-free and model-based reinforcement learning.
• These models offer valuable insights into reward processing and decision-making across species.