Memory for multimodal picture-sound items: Influences of congruency, identifiability and cognitive ageing

Multimodal perception can often evoke performance that is greater than the sum of its unimodal parts in cognitive tasks. For example, choice reaction time research has found responses to multimodal stimuli that are faster than could be explained by responses to its individual unimodal components. This has led to a growth in multimodal perceptual research to explain behavioural observations, but less is known about multimodal influences on more cognitively demanding tasks such as semantic activation and memory. Across 3 well-powered pre-registered memory experiments (total n = 792), this study tested if multimodal encoding could reduce perceptual demands, evoke semantic processing, and encourage episodic recollection. In Experiment 1 the presence of congruent visual information aided later unimodal auditory retrieval (e.g., a dog picture and a barking sound), but multimodal manipulations had similar effects across young and older adults. Experiment 2 equated auditory only and visual only memory for picture-sound items by utilising blurring of the visual pictures, generating a novel shared stimulus set. Here there was no benefit to multimodal encoding, and identifiability of a given stimulus was more strongly related to memory following multimodal encoding than memory following unimodal encoding. Experiment 3 utilised the remember-know paradigm and found no specific multimodal encoding effects on episodic remembering, but did find that item identifiability was more strongly linked to recollection than to familiarity. Our data highlight how multimodal processing is nuanced and supports a general need for multimodal research to explain wholistic sensory processing.