Unfolding spatiotemporal representations of 3D visual perception in the human brain

Although visual input is initially recorded in two dimensions on our retinas, we perceive and interact with the world in three dimensions. Achieving 3D perception requires the brain to integrate 2D spatial representations with multiple depth cues, such as binocular disparity. However, most studies typically examine 2D and depth information in isolation, leaving the integrated nature of 3D spatial encoding largely underexplored. In this study, we collected a large-scale, multimodal neuroimaging dataset consisting of multiple EEG and fMRI sessions while participants viewed stereoscopic 3D stimuli through red-green anaglyph glasses. Participants first completed a behavioral session including depth judgement tasks and a novel cube adjustment task to quantify and calibrate individual depth perception in units of binocular disparity. Then during two EEG and two fMRI sessions, participants passively viewed stimuli presented at 64 systematically sampled 3D locations, yielding over 66,000 trials in total across ten participants. Combining this large-scale EEG-fMRI dataset with computational methods via representational similarity analysis, we not only systematically characterized the spatiotemporal representation of multiple spatial features in 3D perception but also explored how different coordinate systems (e.g., Cartesian or Polar) might be employed across brain regions and time. Our results reveal that human brains employ multiple types of spatial feature representations and coordinate systems to encode spatial locations at different temporal stages and across distinct cortical regions. In addition to strong representations of 2D space throughout visual cortex, we find unique representations for depth and 3D features in later timepoints and visual areas, including some evidence for 3D processing in parahippocampus. These findings contribute to a more comprehensive understanding of the spatiotemporal organization of neural representations that support 3D perception. Additionally, our novel large dataset will be made openly available to support future research on 3D perception and spatial cognition.