Scalable Gaussian process inference of neural responses to natural images

Predicting the responses of sensory neurons is a long-standing neuroscience goal. However, while there has been much progress in modeling neural responses to simple and/or artificial stimuli, predicting responses to natural stimuli remains an ongoing challenge. On the one hand, deep neural networks perform very well on certain datasets but can fail when data are limited. On the other hand, Gaussian processes (GPs) perform well on limited data but are poor at predicting responses to high-dimensional stimuli, such as natural images. Here, we show how structured priors, e.g., for local and smooth receptive fields, can be used to scale up GPs to model neural responses to high-dimensional stimuli. With this addition, GPs largely outperform a deep neural network trained to predict retinal responses to natural images, with the largest differences observed when both models are trained on a small dataset. Further, since they allow us to quantify the uncertainty in their predictions, GPs are well suited to closed-loop experiments, where stimuli are chosen actively so as to collect "informative" neural data. We show how GPs can be used to actively select which stimuli to present, so as to i) efficiently learn a model of retinal responses to natural images, using few data, and ii) rapidly distinguish between competing models (e.g., a linear vs. a nonlinear model). In the future, our approach could be applied to other sensory areas, beyond the retina.