Predictive representations can link model-based reinforcement learning to model-free mechanisms.

Area of Science:

• Neuroscience
• Computational Neuroscience
• Reinforcement Learning

Background:

• Humans and animals use model-based reinforcement learning (RL) to evaluate actions based on future rewards.
• Neural mechanisms underlying model-based RL are not fully understood.
• Evidence suggests overlap between neural circuits for model-based and model-free temporal difference (TD) learning.

Purpose of the Study:

• To propose a framework for model-based computation built upon TD learning.
• To explore the role of the successor representation in bridging model-based and model-free RL.
• To introduce new algorithms that extend this framework.

Main Methods:

• Developing a computational framework integrating the successor representation with TD learning.
• Utilizing simulations to test the behavioral capabilities of the proposed approach.
• Comparing simulation results with observed behaviors in biological organisms.
• Introducing and evaluating two novel algorithms.

Main Results:

• The successor representation combined with TD learning can replicate a subset of model-based behaviors with reduced computational cost.
• Simulations delineated specific behavioral capabilities of this approach.
• New algorithms were developed to address limitations of the successor representation.

Conclusions:

• The proposed framework offers a neurally plausible mechanism for model-based evaluation by integrating with core TD learning.
• This approach accounts for a range of putatively model-based behaviors within a TD learning context.
• The successor representation serves as a foundational element for these neurally plausible algorithms.