Diverse Deep Neural Networks All Predict Human Inferior Temporal Cortex Well, After Training and Fitting.

Area of Science:

• Computational neuroscience
• Cognitive neuroscience
• Machine learning

Background:

• Deep neural networks (DNNs) are leading models for the high-level visual cortex.
• Uncertainty remains regarding the impact of experimental choices (architecture, training, fitting) on DNN-brain similarities.
• Understanding these factors is crucial for advancing computational models of vision.

Purpose of the Study:

• To compare nine DNN architectures in explaining visual representations in the human inferior temporal cortex (hIT).
• To assess the influence of network training and fitting to fMRI data on model performance.
• To investigate how DNNs explain representations in both hIT and primary visual cortex (V1).

Main Methods:

• Compared nine diverse DNN architectures on object recognition tasks.
• Assessed untrained vs. task-trained networks using fMRI data from 62 object images.
• Employed cross-validated fitting to align DNN features with hIT representational dissimilarity matrices.
• Evaluated model performance using independent images and subjects.

Main Results:

• Trained DNNs significantly outperformed untrained models, explaining 57% more variance.
• Model fitting improved DNN-hIT alignment by 124%, indicating features beyond ImageNet training are key.
• The same trained and fitted models explained representations in V1, with earlier layers weighted more heavily.
• All architectures performed comparably after training and fitting.

Conclusions:

• Structured visual features are vital for explaining hIT representations.
• The process of fitting DNNs to brain data reveals crucial information not captured by task training alone.
• The core properties of deep feedforward networks, rather than specific architectures, are key to modeling human visual representations across cortical areas.