Understanding Moment-to-Moment Processing of Visual Narratives

Area of Science:

• Cognitive Psychology
• Computational Linguistics
• Neuroscience

Background:

• Understanding how readers process visual narratives is crucial for effective communication.
• The Scene Perception and Event Comprehension Theory (SPECT) provides a framework for studying these processes.
• Bridging inference generation is a key component of comprehending incomplete information in stories.

Purpose of the Study:

• To investigate the role of inference generation in visual attentional selection during picture story comprehension.
• To test competing hypotheses regarding increased viewing times associated with inference generation: computational load versus visual search.
• To determine if inference-relevant regions in images influence eye movements.

Main Methods:

• Eye-tracking was used to monitor participants' eye movements while they viewed picture stories with manipulated inferable actions.
• Participants' fixation durations and number of fixations were analyzed.
• A separate group rated the inferential-relevance of specific image regions.

Main Results:

• Participants made more fixations, rather than having longer fixation durations, when generating inferences, starting around the fifth fixation.
• Inferentially relevant regions within critical images predicted differences in viewers' eye movements.
• Viewer's event models in working memory influence visual attention during narrative processing.

Conclusions:

• Inference generation during picture story reading primarily affects visual search patterns (more fixations) rather than increasing cognitive load (longer fixations).
• Specific image regions crucial for inference guide attentional selection.
• Working memory event models play a significant role in directing visual attention in dynamic visual narratives.