Value-dependent selection in the brain: simulation in a synthetic neural model

Many forms of learning depend on the ability of an organism to sense and react to the adaptive value of its behavior. Such value, if reflected in the activity of specific neural structures (neural value systems), can selectively increase the probability of adaptive behaviors by modulating synaptic changes in the circuits relevant to those behaviors. Neuromodulatory systems in the brain are well suited to carry out this process since they respond to evolutionarily important cues (innate value), broadcast their responses to widely distributed areas of the brain through diffuse projections, and release substances that can modulate changes in synaptic strength. The main aim of this paper is to show that, if value-dependent modulation is extended to the inputs of neural value systems themselves, initially neutral cues can acquire value. This process has important implications for the acquisition of behavioral sequences. We have used a synthetic neural model to illustrate value-dependent acquisition of a simple foveation response to a visual stimulus. We then examine the improvement that ensues when the connections to the value system are themselves plastic and thus become able to mediate acquired value. Using a second-order conditioning paradigm, we demonstrate that auditory discrimination can occur in the model in the absence of direct positive reinforcement and even in the presence of slight negative reinforcement. The discriminative responses are accompanied by value-dependent plasticity of receptive fields, as reflected in the selective augmentation of unit responses to valuable sensory cues. We then consider the time-course during learning of the responses of the value system and the transfer of these responses from one sensory modality to another. Finally, we discuss the relation of value-dependent learning to models of reinforcement learning. The results obtained from these simulations can be directly related to various reported experimental findings and provide additional support for the application of selectional principles to the analysis of brain and behavior.