Meta-reinforcement learning via orbitofrontal cortex.

Area of Science:

• Neuroscience
• Computational Neuroscience
• Cognitive Science

Background:

• Meta-reinforcement learning (meta-RL) enhances generalization in deep RL models.
• The prefrontal cortex is hypothesized to mediate brain-based meta-RL, but evidence is limited.

Purpose of the Study:

• To investigate the role of the orbitofrontal cortex (OFC) in mediating meta-RL.
• To elucidate the neural mechanisms and timescales of RL algorithms in the OFC.

Main Methods:

• Trained mice and deep RL models on a probabilistic reversal learning task.
• Investigated the necessity of Ca2+/calmodulin-dependent protein kinase II-dependent synaptic plasticity in OFC for meta-learning.
• Monitored OFC activity and assessed behavioral impact of OFC inactivation.

Main Results:

• OFC mediates meta-RL, with synaptic plasticity being essential for meta-learning, not for expert within-session RL.
• Post-meta-learning, OFC activity robustly encoded value signals, and OFC inactivation impaired RL behaviors.
• Longitudinal tracking showed meta-learning gradually shapes OFC population value coding to guide behavior.

Conclusions:

• The OFC implements meta-RL, utilizing distinct neural mechanisms and timescales.
• Two coexisting RL algorithms within the OFC support adaptive decision-making.