Timing Matters: Cross-Codal Integration of Bridging Event Information Happens While Viewing Picture Stories

Despite the prevalence of multi-codal narratives-such as comics consisting of pictures and text-in everyday life, little is understood about how individuals integrate information across different codalities to form cohesive mental representations. In the present study, we explored when processing costs emerged during cross-codal integration of bridging event information in multi-codal narratives. Participants were presented with uni-codal (i.e., consisting only of pictures or of text) and multi-codal stories in which the bridging event information was presented in the other codality. We measured viewing times for the bridging event and end-state panels to reflect the cognitive costs associated with integration. The results showed a significant interaction between codality switch and panel position. While the presence of a codality switch increased viewing times for both panel positions, there was a much larger increase for the bridging event panel than the end-state panel. This result supports a hypothesis suggesting that during narrative understanding, information of a different codality is integrated into the mental representation of the narrative upon encountering it. Our results underscore the influence of codality on narrative integration, indicating that understanding may be affected by the format in which information is presented. Future models of event cognition should incorporate codality as a factor in narrative comprehension.