Temporal asymmetry of neural pattern similarity predicts recognition memory decisions

A stimulus can be familiar for multiple reasons. It might have been recently encountered, is similar to recent experience, or is similar to 'typical' experience. Understanding how the brain translates these sources of similarity into memory decisions is a fundamental, but challenging goal. Here, using functional magnetic resonance imaging (fMRI), we computed neural similarity between a current stimulus and events from different temporal windows in the past and future (from seconds to days). We show that trial-by-trial recognition memory decisions (is this stimulus 'old'?) were predicted by the difference in similarity to past vs. future events (temporal asymmetry). This relationship was (i) evident in lateral parietal and occipitotemporal cortices, (ii) strongest when considering events from the recent past (minutes ago), and (iii) most pronounced when veridical (true) memories were weak. These findings elucidate how the brain evaluates past experience in service of making recognition memory decisions.