A multimodal encoding model applied to imaging decision-related neural cascades in the human brain

Perception and cognition in the brain are naturally characterized as spatiotemporal processes. Decision-making, for example, depends on coordinated patterns of neural activity cascading across the brain, running in time from stimulus to response and in space from primary sensory regions to the frontal lobe. Measuring this cascade is key to developing an understanding of brain function. Here we report on a novel methodology that employs multi-modal imaging for inferring this cascade in humans at unprecedented spatiotemporal resolution. Specifically, we develop an encoding model to link simultaneously measured electroencephalography (EEG) and functional magnetic resonance imaging (fMRI) signals to infer high-resolution spatiotemporal brain dynamics during a perceptual decision. After demonstrating replication of results from the literature, we report previously unobserved sequential reactivation of a substantial fraction of the pre-response network whose magnitude correlates with a proxy for decision confidence. Our encoding model, which temporally tags BOLD activations using time localized EEG variability, identifies a coordinated and spatially distributed neural cascade that is associated with a perceptual decision. In general the methodology illuminates complex brain dynamics that would otherwise be unobservable using fMRI or EEG acquired separately.