Neuronal Encoding in Prefrontal Cortex during Hierarchical Reinforcement Learning.

Area of Science:

• Neuroscience
• Computational Neuroscience
• Cognitive Science

Background:

• Reinforcement learning (RL) models excel at explaining learning but struggle with real-world complexity.
• Hierarchical RL addresses this by structuring behavior into temporally abstract actions (options) linked to subgoals.
• Pseudoreward prediction errors, distinct from final reward prediction errors, arise from subgoal attainment and correlate with ACC activation in humans.

Purpose of the Study:

• To investigate the single-neuron encoding of pseudoreward prediction errors in the anterior cingulate cortex (ACC).
• To determine the role of ACC in hierarchical reinforcement learning by examining neural activity during a subgoal-based task.

Main Methods:

• Trained two animals on a primate version of a task designed to elicit pseudoreward prediction errors.
• Recorded single-unit electrical activity in the ACC, lateral prefrontal cortex, and orbitofrontal cortex (OFC) during task performance.

Main Results:

• A small population of neurons in the ACC exhibited firing rates encoding pseudoreward prediction errors.
• These specific neurons were exclusively found within the ACC.
• The number of neurons encoding pseudoreward prediction errors was notably smaller than those encoding information about the main task goal.

Conclusions:

• The findings support the role of the ACC in encoding subgoals and pseudoreward prediction errors for hierarchical reinforcement learning.
• ACC neurons appear critical for processing intermediate success signals within complex behavioral sequences.
• The limited population size of these neurons suggests specialized processing for subgoal-related information within the ACC.