When knowledge interferes with perception: Neural mechanisms of the semantic amplification of visual false memory

Visual false memory refers to our tendency to falsely recognize novel stimuli that are visually similar to seen stimuli. Visual false memory also occurs when stimuli are meaningful, suggesting that semantic information interferes with the encoding of visual details. However, the neural mechanisms of this semantic interference effect are largely unknown. In the present fMRI study, participants were scanned while encoding visually similar fonts presented with words (word-fonts) or pseudowords (pseudoword-fonts), and later, when recognizing old, new similar (lures), and new dissimilar (novel) fonts displayed in the same meaningless letter string. We performed (1) representational similarity analysis (RSA) at encoding to identify visual, visuosemantic, and semantic representations associated with subsequent visual true and false font recognition, (2) encoding-retrieval similarity (ERS) analysis to assess their reinstatement during retrieval, and (3) mediational analyses to examine hippocampal contributions. The study yielded three main findings. First, visuosemantic representations supported true font recognition when stored in right fusiform gyrus, but false recognition of word-fonts when stored in the left fusiform gyrus. Second, mirroring this pattern, reinstatement in right fusiform gyrus was associated with true font recognition, whereas reinstatement in left fusiform gyrus was linked to false recognition of word-fonts. Finally, posterior hippocampal activation reduced false font memory mainly for pseudoword-associated fonts via decreased reinstatement in perceptual regions, while anterior hippocampal activity increased false memory of word-fonts via enhanced reinstatement in semantic regions. Taken together, these findings reveal how distinct hippocampal-cortical pathways differentially bias memory towards perceptual specificity or semantic generalization.