Semantic association of ecologically unrelated synchronous audio-visual information in cognitive integration: an...

In this paper, we aimed to study the semantic association of ecologically unrelated synchronous audio-visual information in cognitive integration. A moving particle, which speed varied, was taken as a visual stimulus, while a simple tone, which frequency varied, was used as an auditory stimulus, both were synchronously presented to subjects in the form of a video. Behavioral results confirmed our hypothesis that the moving particle with varied speed and the simple tone with varied frequency were highly associated. Event-related potential (ERP) results showed that an N400 effect and a late posterior negativity (LPN) were elicited under the Incongruent condition as compared to the Congruent condition. It was further determined that there was semantic association between ecologically unrelated synchronous audio-visual information in cognitive integration. We considered that the N400 effect in our results reflected the process that stimulus-driven activities are bound together through a temporal semantic network (TSN) to form multimodal representations, while the state of this temporal semantic network was determined by both long-term learned association among stimuli and short-term experience of incoming information. The LPN might reflect the process that the human brain searches and retrieves context-specifying information in order to make a judgment, and the context-specifying information might have originated from the long-term learned association stored in the brain.