What is dopamine doing in model-based reinforcement learning?

Area of Science:

• Neuroscience
• Computational Neuroscience
• Reinforcement Learning

Background:

• Dopamine's role in reinforcement learning (RL) is well-established, primarily linked to reward prediction errors (RPEs) in model-free RL.
• However, recent experiments suggest dopamine's involvement in model-based RL, a finding that challenges existing theories.
• This discrepancy necessitates a closer examination of dopamine's function in complex learning paradigms.

Purpose of the Study:

• To investigate the mechanisms underlying dopamine's role in model-based reinforcement learning (RL).
• To evaluate two competing hypotheses explaining dopamine's involvement in model-based RL.
• To reconcile the known function of dopamine in model-free RL with its emerging role in model-based RL.

Main Methods:

• The study theoretically examines two potential explanations for dopamine's function in model-based RL.
• Hypothesis 1: Dopamine neurons encode a prediction error for updating the successor representation.
• Hypothesis 2: Combined effects of RPEs and surprise signals explain dopamine's involvement.

Main Results:

• The study provides a theoretical framework for understanding dopamine's function in model-based RL.
• It explores how dopamine might contribute to learning predictive state representations (successor representation).
• It considers the interplay between reward prediction errors and surprise signals in dopaminergic activity.

Conclusions:

• Dopamine's involvement in model-based RL can be explained through its role in updating predictive representations or through a combination of RPEs and surprise.
• These findings offer a more comprehensive understanding of dopamine's multifaceted role in reinforcement learning.
• Further experimental validation is needed to distinguish between the proposed mechanisms.