Modulation of associative learning in the hippocampal-striatal circuit based on item-set similarity

Mounting evidence suggests that the medial temporal lobe (MTL) and striatal learning systems support different forms of learning, which can be competitive or cooperative depending on task demands. We have previously shown how activity in these regions can be modulated in a conditional visuomotor associative learning task based on the consistency of response mappings or reward feedback (Mattfeld & Stark, 2015). Here, we examined the shift in learning towards the MTL and away from the striatum by placing strong demands on pattern separation, a process of orthogonalizing similar inputs into distinct representations. Mnemonically, pattern separation processes have been shown to rely heavily on processing in the hippocampus. Therefore, we predicted modulation of hippocampal activity by pattern separation demands, but no such modulation of striatal activity. Using a variant of the conditional visuomotor associative learning task that we have used previously, we presented participants with two blocked conditions: items with high and low perceptual overlap during functional magnetic resonance imaging (fMRI). As predicted, we observed learning-related activity in the hippocampus, which was greater in the high than the low overlap condition, particularly in the dentate gyrus. In contrast, the associative striatum also showed learning related activity, but it was not modulated by overlap condition. Using functional connectivity analyses, we showed that the correlation between the hippocampus and dentate gyrus with the associative striatum was differentially modulated by high vs. low overlap, suggesting that the coordination between these regions was affected when pattern separation demands were high. These findings contribute to a growing literature that suggests that the hippocampus and striatal network both contribute to the learning of arbitrary associations that are computationally distinct and can be altered by task demands.