Enhancing reinforcement learning models by including direct and indirect pathways improves performance on striatal...

Area of Science:

• Neuroscience
• Computational Neuroscience
• Machine Learning

Background:

• Reward learning is dopamine-dependent, involving synaptic plasticity in cortico-striatal pathways.
• Existing reinforcement learning models use reward prediction error but struggle with complex goal-directed behaviors like renewal and reversal.

Purpose of the Study:

• To present TD2Q, a novel reinforcement learning model that better represents basal ganglia circuitry.
• To enhance the simulation of animal learning behaviors and understand the roles of direct and indirect pathways.

Main Methods:

• Developed TD2Q, a two-Q matrix model (G for direct, N for indirect pathway neurons) updated via temporal difference reward prediction error.
• Tested TD2Q on multi-step tasks including extinction, renewal, discrimination, reward probability switching, and sequence learning.

Main Results:

• TD2Q simulations closely matched rodent behaviors in choice and sequence learning tasks.
• Temporal difference reward prediction error was essential for multi-step task learning.
• Blocking the N matrix update impaired discrimination learning, aligning with experimental findings.
• Sequence learning performance significantly improved with the two-matrix structure.

Conclusions:

• Incorporating basal ganglia physiology into reinforcement learning models enhances performance and behavioral mimicry.
• TD2Q provides insights into the functional roles of direct and indirect striatal pathway neurons in learning.