During natural vision, semantic novelty modulates fixation-related processing in primate cortex

We sample visual scenes with short gaze fixations separated by saccades. While low-level integration is known, semantic integration of foveal vision across fixations remains unclear. We hypothesized that the brain responds to changes in semantic information from one fixation to the next, and therefore postulated a neural signal associated with semantic novelty for each saccade. Novelty was measured from different layers of a deep network on foveal vision. Only semantic, and not low- or mid-level, novelty modulated frontal and occipital fixation-related potentials in male and female human scalp EEG during natural viewing of full-length movies (3.4x10⁶ saccades). Intracranial recordings in male and female humans (9.0x10⁴ saccades) and female non-human primates (Macaca mulatta; 3.3x10⁴ saccades) revealed broadband high-frequency activity modulations in scene-specific ventral visual and frontal brain areas only for semantic novelty. This modulation was stronger for movies than static images, and frontal modulation occurred early alongside occipital modulation, suggesting top-down effects. This modulation of fixation-related activity with novelty suggests that foveal representations are compared between consecutive fixations to construct scene representations during natural viewing in primates.Significance statement Primate vision relies on eye movements to sample the world, yet how the brain integrates high-level meaning across these "snapshots" remains a mystery. By combining deep-learning models with massive electrophysiological datasets from humans and macaques, we show that the brain responds to changes in semantic content between fixations, but not low- or mid-level changes. We identified a "semantic novelty" signal that modulates neural activity across the primate cortex. These findings suggest visual processing is not a series of independent glimpses, but a continuous process comparing consecutive foveal information at the semantic level. This work bridges biological vision and artificial intelligence, providing a new framework for understanding how the brain maintains a coherent understanding of a dynamic world.