Crossmodal Semantic Congruence Interacts with Object Contextual Consistency in Complex Visual Scenes to Enhance...

Area of Science:

• Cognitive Psychology
• Auditory Perception
• Visual Memory

Background:

• Object sounds can enhance visual perception.
• The impact of crossmodal semantic congruence on post-perceptual stages like short-term memory (STM) is not well understood.
• The role of visual scene context in modulating crossmodal effects on memory requires investigation.

Purpose of the Study:

• To investigate if crossmodal semantic congruence affects short-term memory (STM) for visual stimuli.
• To determine if the visual scene context modulates the effect of object sounds on visual STM.
• To examine the influence of auditory cues on the spatial recall of visual targets.

Main Methods:

• Two experiments were conducted using visual scenes and object sounds.
• Participants performed a spatial recall task for a visual target object after a delay.
• Auditory stimuli were either semantically congruent or incongruent with the visual target, and the target was presented in- or out-of-context within the scene.

Main Results:

• Greater accuracy was observed in judging the spatial position of targets when paired with semantically congruent object sounds.
• The facilitatory effect of object sounds on spatial recall was reduced when the target object was out-of-context within the visual scene.
• Crossmodal facilitation was dependent on the semantic consistency between the target object and its background scene.

Conclusions:

• Object sounds facilitate the selection and processing of semantically related visual stimuli, impacting short-term memory.
• The effectiveness of crossmodal semantic facilitation in memory tasks is contingent upon the visual scene's semantic configuration.
• Auditory-visual semantic congruence enhances memory, but this effect is diminished by contextual incongruity.