An anatomical substrate of credit assignment in reinforcement learning

A key problem in learning is credit assignment. Biological systems lack a plausible mechanism to implement the backpropagation approach, a method that underlies much of the dramatic progress in artificial intelligence. Here, we use automated connectomic analysis to show that the synaptic architecture of songbird basal ganglia (Area X) supports local credit assignment using a variant of a node perturbation algorithm proposed in a model of reinforcement learning. Using two volume electron microscopy (vEM) datasets, we find that key predictions of the model hold true: axons that encode exploratory variability terminate predominantly on dendritic shafts, while axons that encode song timing (context) terminate predominantly on spines. Based on the detailed EM data, we then built a biophysical model of reinforcement learning that suggests that the synaptic dichotomy between variability and context encoding axons facilitates efficient learning. In combination, these findings provide strong evidence for a general, biologically plausible credit assignment model in vertebrate basal ganglia learning.