Eligibility-trace-gated depression in predecessor feature learning enables post-reward exploration

Area of Science:

• Neuroscience
• Computational Neuroscience
• Reinforcement Learning

Background:

• Dopamine's role in reward prediction error is established in reinforcement learning.
• Computational functions of other neuromodulators, like acetylcholine (ACh), are less understood.
• Flexible learning and exploration are crucial for adapting to changing environments.

Purpose of the Study:

• Investigate the effect of ACh modulation on predecessor feature (PF)-based learning.
• Develop an ACh-modulated PF algorithm (ACh-PF) using synaptic depression to promote exploration.
• Compare ACh-PF performance against a baseline PF algorithm in varying maze complexities.

Main Methods:

• Developed an ACh-modulated PF algorithm (ACh-PF) implementing synaptic depression.
• Utilized eligibility-trace outer products for synaptic depression in the ACh-PF model.
• Tested algorithms in n-arm radial mazes with conventional and post-reward exploration criteria.

Main Results:

• ACh-PF demonstrated improved performance under a post-reward exploration criterion compared to baseline PF.
• ACh-PF performance showed a non-monotonic dependence on the synaptic depression parameter, with an optimal intermediate regime.
• Performance degraded with increased spatial and action-space complexity, revealing scaling limits.

Conclusions:

• Cholinergic-like synaptic depression is linked to flexible exploration in spatial navigation.
• The ACh-PF algorithm offers a computational framework for understanding ACh's role in exploration.
• Environmental complexity imposes scaling limits on the efficiency of exploration strategies.