Narrative context shifts gaze from visual to semantic salience

Area of Science:

• Cognitive Science
• Computational Neuroscience
• Psychology

Background:

• Human eye movements are crucial for visual information gathering, exceeding 100,000 daily.
• Gaze behavior is influenced by both visual features and cognitive goals, aiming for understanding rather than mere representation.
• Information updating understanding should attract more attention than purely visually salient elements.

Purpose of the Study:

• To investigate whether semantic saliency, reflecting understanding, or visual saliency better predicts gaze behavior.
• To test if coherent temporal context enhances attention to semantically salient information compared to shuffled context.
• To evaluate the utility of large language models in quantifying semantic saliency for gaze analysis.

Main Methods:

• A free-viewing paradigm using image narratives presented in coherent or temporally shuffled orders.
• Development of a computational model to quantify semantic saliency using language narratives and contextual surprisal from a large language model.
• Comparison of the semantic saliency model's predictive power against a state-of-the-art visual saliency model (DeepGaze-II).

Main Results:

• Participants looked more often and faster at semantically salient objects in coherent narratives compared to shuffled ones.
• Visual saliency did not show a differential effect on gaze behavior between coherent and shuffled narrative orders.
• The semantic saliency model demonstrated a significant ability to explain gaze behavior in meaningful contexts.

Conclusions:

• Internal cognitive models, informed by context, actively guide visual attention.
• Semantic saliency, quantifiable via language models, is a key determinant of gaze behavior in understanding-driven tasks.
• Language models offer a promising approach for analyzing gaze in complex, meaningful visual environments.