Sound Flexibly Interacts With Dual-Format Templates to Optimize Visual Search

Previous studies indicate that the dual formats of templates recursively operate in tandem during visual search tasks. The guiding template with coarse features supports target localization, whereas the target template with precise features supports target identification. However, whether a sound that provides target-consistent identity information would flexibly interact with these two templates to optimize search behavior remains unclear. Across two experiments, event-related potentials (ERPs) were recorded while participants searched for and identified a visual target letter ("V" or "K"). In Experiment 1, visual targets were presented in isolation (V) or synchronously with a speech sound that was congruent (AVcon) or incongruent (AVincon) with the target; Experiment 2 additionally paired the target with a sound unrelated to both potential targets and distractors (AVun). Behaviorally, converging findings across experiments indicated that relative to AVun and V conditions, congruent sounds improved accuracy and shortened response times, whereas incongruent sounds prolonged response times. ERP results revealed early cross-modal interactions, reflected in amplitude modulations of the occipital P195 and centroparietal N270 components in the AVcon and AVincon conditions relative to summed unimodal ERPs (Exp. 1); these interactions were reduced in the AVun condition, in which the sound was uninformative about potential targets (Exp. 2). Cross-condition multivariate pattern analysis (MVPA) showed higher target-presence decoding accuracy in the AVcon condition than in the AVincon, AVun, and V conditions. Peak-to-peak improvements in the AVcon-AVincon and AVcon-V comparisons, but not the AVcon-AVun comparison, were positively correlated with corresponding improvements in behavioral accuracy. These findings suggest that the interaction between sounds and the dual formats of templates is contingent on the usefulness of available information. When the guiding template is relatively efficient, task-irrelevant sounds may generally interact with the entire visual search array. Subsequently, congruent sounds selectively enhance target-template processing, potentially contributing to improved target identification.