Reward prediction errors, not sensory prediction errors, play a major role in model selection in human reinforcement...

Area of Science:

• Neuroscience
• Cognitive Science
• Computational Neuroscience

Background:

• Model-based reinforcement learning (RL) is crucial for adapting to dynamic environments.
• The brain's mechanism for selecting appropriate internal models during RL remains poorly understood.
• Current theories propose either sensory prediction errors or reward prediction errors drive model selection.

Purpose of the Study:

• To investigate the neural mechanisms of internal model selection in human reinforcement learning.
• To compare the predictive power of sensory prediction error-based versus reward prediction error-based models of human behavior.
• To determine whether the brain utilizes sensory or reward prediction errors for selecting environmental models.

Main Methods:

• Designed a switching experiment transitioning between first-order and second-order Markov decision processes.
• Tested two computational models: a sensory prediction-error-driven Bayesian algorithm and a reward-prediction-error-driven policy gradient algorithm.
• Compared model simulations against human behavioral data from the RL task.

Main Results:

• Model fitting analyses indicated that the policy gradient algorithm provided a better fit to human RL behavior than the Bayesian algorithm.
• This suggests that reward prediction errors play a more significant role in model selection than sensory prediction errors.
• Human participants' behavior aligned more closely with predictions from the reward-prediction-error-driven model.

Conclusions:

• The human brain likely employs reward prediction errors to select appropriate internal models during reinforcement learning.
• This finding challenges the primacy of sensory prediction errors in internal model selection.
• The study provides evidence for a reward-based mechanism underlying cognitive flexibility in RL tasks.