Steerable autoencoders underlying remapping, spatiotopy, and visual stability

The encoding of locations in world coordinates is often called spatiotopy; it captures how we experience the layout of things around us. The visual system, on the other hand, appears to rely on retinotopic representations. Here we review earlier suggestions that replace spatiotopic maps with the tracking or remapping of a small number of attended targets to keep their positions updated on retinotopic maps, combined with suppression of motion responses to the shift of the retinal image. Importantly, we propose that each attended target's identity is connected to the locations of its features in early visual cortex through a steerable autoencoder network that shifts the feature activity to the target's new location with each saccade, explaining the many reports of trans-saccadic integration. This autoencoder process is part of the overall architecture of the visual system, acting to link target properties together, even as the target or the eyes move.