Correctness is its own reward: bootstrapping error signals in self-guided reinforcement learning

Area of Science:

• Neuroscience
• Computational Neuroscience
• Animal Behavior

Background:

• Reinforcement learning (RL) typically requires external reward functions for skill acquisition.
• Self-directed learning, like zebra finch song learning, challenges RL by lacking explicit external guidance.
• The neural basis for internally generated performance assessment in song learning remains unclear.

Purpose of the Study:

• To propose and model a unified neural circuit for tutor song memorization and performance assessment.
• To investigate how predictive cancellation of tutor song can generate error signals for self-directed learning.
• To demonstrate that these error signals can train a reinforcement learning agent.

Main Methods:

• Developed computational models of a local forebrain circuit involved in song learning.
• Simulated learning through synaptic plasticity, focusing on predictive cancellation of auditory tutor song input.
• Analyzed neural population activity for error coding and modeled the learning process in stages.

Main Results:

• Learned circuits generated population error codes signaling performance mismatches.
• Anti-Hebbian plasticity in inhibitory interneurons best explained experimental data.
• Model learning occurred in two distinct phases: error sensitivity sharpening and response minimization.
• The generated error signal successfully trained a reinforcement learning agent to replicate bird songs.

Conclusions:

• A single neural circuit using predictive cancellation can subserve both song memory and performance assessment.
• Local learning rules are sufficient to generate error signals for self-directed behavioral learning.
• This mechanism provides a potential explanation for how animals learn complex behaviors without external rewards.