Dynamic cognitive processes of text-picture integration revealed by event-related potentials

The integration of text and picture is the core of multimedia information processing. Relevant theories suggest that text and picture are processed through different channels in the early stage, and integrated in the late stage of processing. Based on these theories, the current study adopted measures of event-related potentials to examine the cognitive and neural processes of text-picture integration. The results showed that in the early stage of text-picture integration, picture processing evoked a more negative N1 over the occipital area and a N300 over the prefrontal area, which might reflect the discrimination process of visual stimuli and the imagery representation of the picture, respectively; in the late stage, the text-picture induced a N400 in the central area and an LPC over the central, parietal and temporal areas, which might be associated with the semantic activation and integration of text and picture, respectively. These results not only provide support for existing theories, but also further elucidate the dynamic neural processing of text-picture integration in terms of its temporal and spatial characteristics.