Understanding phototexts through an eye-tracking study on visual and emotional interactions between text and photo

Visual perception is a complex cognitive process that allows humans to extract meaningful information from their environment. Phototexts, artistic and literary compositions that integrate photographic and textual elements, offer a multimodal experience that requires readers to navigate between visual and linguistic modes of interpretation. This study investigates, in naïve participants, the influence of spatial layout configuration, emotional content, enjoyment, and attentional focus on the visual exploration of phototexts. Using eye-tracking technology, we examined participants' gaze patterns, focusing on metrics such as first fixation latency and fixation distribution across areas of interest (AOIs). Additionally, we explored the presence of left-gaze bias in text-photo stimuli systematically counterbalanced to mitigate positional effects. Our findings reveal key aspects of oculomotor behavior, as well as explicit preferences and judgments regarding phototext perception. Eye-tracking data indicated a natural reading bias, with shorter fixation latencies for text regardless of position, alongside a left-gaze bias that emerged specifically for photographs. Interestingly, attention patterns evolved dynamically over time, with an initial focus on text shifting toward the photographic component as viewing progressed. By analyzing phototexts as complex multimodal stimuli, this study provides novel insights into their synergistic effects on attention distribution and perceptual engagement.