A working memory model based on recurrent neural networks using reinforcement learning

Numerous electrophysiological experiments have reported that the prefrontal cortex (PFC) is involved in the process of working memory. PFC neurons continue firing to maintain stimulus information in the delay period without external stimuli in working memory tasks. Further findings indicate that while the activity of single neurons exhibits strong temporal and spatial dynamics (heterogeneity), the activity of population neurons can encode spatiotemporal information of stimuli stably and reliably. From the perspective of neural networks, the computational mechanism underlying this phenomenon is not well demonstrated. The main purpose of this paper is to adopt a new strategy to explore the neural computation mechanism of working memory. We used reinforcement learning to train a recurrent neural network model to learn a spatial working memory task. The model is composed of a decision network and a baseline network. The decision network is responsible for updating strategies to make action choices, while the baseline network evaluates action choices to predict rewards. Simulated results demonstrate that the model can perform the spatial working memory task. The activity of the recurrent units has characteristics such as temporal dynamics and preferred direction selectivity, but their population activity encodes the stimulus information stably during the delay period in a low-dimensional subspace. These activity characteristics displayed by the model units are similar to those of PFC neurons observed in the same experiments. Meanwhile, as the network model continued learning the task, the temporal stability and spatial separability of the stimulus information encoded by the activity of model units in the low-dimensional subspace gradually strengthened, and the accuracy of the network's action choices also increased. In summary, this network model provides a new simulation method for spatial working memory tasks and a new perspective for understanding the characteristics of neuron activity in the PFC.