Brain signals of a Surprise-Actor-Critic model: Evidence for multiple learning modules in human decision making

Area of Science:

• Neuroscience
• Cognitive Science
• Computational Neuroscience

Background:

• Human reward learning over extended action sequences is complex, potentially involving parallel neural modules.
• Existing brain imaging methods face limitations in experimental paradigms, signal dissociation, and computational model scope.

Purpose of the Study:

• To overcome limitations in studying learning modules using functional magnetic resonance imaging (fMRI).
• To introduce a complex sequential decision-making task with surprising events to differentiate learning signals.
• To test a wide range of reinforcement learning algorithms, including a novel surprise-modulated approach.

Main Methods:

• Developed a complex sequential decision-making task incorporating surprising events.
• Utilized fMRI to measure brain activity (blood oxygen level dependent - BOLD responses).
• Tested behavior against numerous model-free, model-based, and hybrid reinforcement learning algorithms, including a novel surprise-modulated actor-critic model.

Main Results:

• Action choices were best explained by model-free policy gradient, but reaction times and neural data were not fully accounted for.
• Identified distinct neural signatures for both model-free learning and surprise-based learning in fMRI BOLD responses.
• Demonstrated that surprise, derived from state prediction error, modulates the learning rate of a model-free actor.

Conclusions:

• The brain likely employs multiple parallel learning modules, as evidenced by distinct reward prediction error and surprise signals.
• Findings extend previous fMRI research on learning to multi-step decision-making scenarios.
• Emphasized the significant roles of policy gradient and surprise signaling in human learning processes.