Novelty is not surprise: Human exploratory and adaptive behavior in sequential decision-making.

Area of Science:

• Cognitive Neuroscience
• Computational Neuroscience
• Behavioral Economics

Background:

• Classic reinforcement learning (RL) theories struggle to explain human decision-making under conditions of no external reward or environmental volatility.
• Human behavior often deviates from standard RL predictions when facing sparse rewards and abrupt environmental shifts.

Purpose of the Study:

• To develop an enhanced RL framework that accounts for human behavior in complex environments.
• To investigate the distinct roles of novelty and surprise in guiding human exploration and learning.

Main Methods:

• Utilized a deep sequential decision-making paradigm with sparse rewards and abrupt environmental changes.
• Developed a computational model integrating novelty and surprise into RL algorithms.
• Recorded electroencephalography (EEG) signals to analyze neural correlates of surprise, novelty, and reward.

Main Results:

• Novelty was found to drive exploration prior to initial reward discovery.
• Surprise was shown to accelerate the learning of both world-models and model-free action-values.
• Human decisions were predominantly driven by model-free action choices, with limited use of the world-model for planning but significant role in detecting surprising events.
• The proposed theory accurately predicted human action choices and allowed for dissociation of surprise, novelty, and reward in EEG data.

Conclusions:

• Integrating surprise and novelty into RL theories is crucial for explaining human behavior in dynamic and unrewarded environments.
• While humans utilize model-free strategies, the world-model plays a key role in identifying surprising events, which in turn influences learning.
• The developed framework offers a more comprehensive understanding of decision-making and provides a neural basis for differentiating key learning signals.