Cross-codal bridging information processing by L1 Chinese speakers: An ERP study

Using a rapid serial visual presentation (RSVP) paradigm, this study examined how Chinese first language readers process codality switch in reading comprehension. 28 participants read 160 four-segment pictorial or textual stories where the second segment containing the bridging information is manipulated to achieve four conditions, namely, (1) picture-unswitched (stories comprising four pictures), (2) picture-switched (pictorial stories where the second panel is described in a sentence), (3) text-unswitched (stories comprising four sentences), and (4) text-switched (textual stories where the second sentence is depicted in a picture). The findings demonstrated that codality switch does not impose additional cognitive load in narrative processing, and revealed both codality-specific processing at early stages of visual feature processing as well as a shared conceptual representation system at later stages. Individual differences in visuospatial working memory and visual language fluency predicted behavioural accuracy, but not ERP amplitudes. These findings provide empirical support for the Multimodal Parallel Architecture (Cohn & Schilperoord, 2022), suggesting that during cross-codal processing, codality-specific perceptual processes lead to a shared conceptual system across codalities.