Monkey upload: Improving robustness using multi-stage neural alignment

Recent research has shown that aligning neural network representations with neural recordings has the effect of regularizing them (Federer, Xu, Fyshe, & Zylberberg, 2020; Li et al., 2019; Pirlot, Gerum, Efird, Zylberberg, & Fyshe, 2022), improving their performance across several metrics. However, previous work in this field has predominantly used neural data from the V1 cortex, and comparable performance gains are obtained when the V1 data are replaced by a noise distribution with similar statistics (Federer et al., 2020; Pirlot et al., 2022). In this research, we present the results of aligning the hidden layers of a convolutional neural network model with macaque brain recordings from V1, V4, and IT. Previous research, using only data from the inferior temporal cortex (Dapello et al., 2022), resulted in improved robustness to adversarial examples. Here we show that aligning all three areas improves the model's robustness against several different kinds of corruption. Furthermore, we show that the standard alignment approach, deep canonical correlation analysis (DCCA) is not necessary to achieve good results. We found that when using the VICReg unsupervised loss function for alignment, the model displays more reliable robustness against a subset of corruptions than DCCA. When aligning all three areas, VICreg is superior to InfoNCE (Oord, Li, & Vinyals, 2019) and DCCA, which show robustness to 4, 2, and 0 distortions, respectively. Finally, we show that DCCA is very sensitive to randomizing the data, VICReg is mildly affected, and InfoNCE is not sensitive at all. Hence, for two of the models, it is not just the distribution of the data that matters when aligning all three areas. Overall, our research provides further support for the impact of neural data in developing more robust and neurobiologically plausible models of vision.