The spatiotemporal neural dynamics of action-related features underlying action recognition

The lateral occipitotemporal cortex (LOTC) has been suggested to host action representations that are thought to contribute to accessing meaning of observed actions. In line with this proposal, previous studies have shown spatially overlapping representations of various action-related features including objects, scenes, social properties, and kinematics, as well as abstract representations pertaining to action semantics. Less is known, however, about the way in which these features are integrated during action recognition. To address this question, we aimed to examine the temporal order in which different action-related features emerge in the EEG signal using EEG-based representational similarity analysis (RSA). Additionally, to investigate the spatiotemporal characteristics of these neural representations, we employed fMRI-EEG fusion. Static images spanning 27 everyday actions (e.g., riding a bike, washing dishes, brushing hair) were shown to participants (N = 24) in a delayed matching task, in a total of 648 trials (using different combinations of locations, actors, and viewpoints). Participants were asked to judge stimuli with respect to the type of action, location, and actor identity. Temporally specific neural representational dissimilarity matrices (RDMs) pertaining to neural representations of action categories were generated from 64-channel EEG recordings using time-resolved RSA. These were correlated with behavioral RDMs pertaining to semantic action (dis)similarities, revealing a time course of neural semantic action representations. Additionally, we considered RDMs capturing (dis)similarities between actions in terms of involved body postures, scenes, social aspects, and lower-level visual information. Results suggest a temporally ordered hierarchical buildup of neural representations related to visual, contextual, body-related, and semantic action information. fMRI-EEG fusion analysis further suggests that striate and extrastriate areas along the lateral visual pathway encode lower-level visual and body-related properties of actions, and that contextual and semantic information is integrated in the LOTC. These outcomes provide a spatiotemporal characterization of the neural processes enabling humans to recognize goal-directed actions.