Quantifying Task-relevant Similarities in Representations Using Decision Variable Correlations

Area of Science:

• Neuroscience
• Artificial Intelligence
• Computer Vision

Background:

• Previous research has explored similarities between neural activity in the visual cortex and deep neural networks (DNNs).
• Existing studies present conflicting findings regarding the degree of representational similarity between biological vision and DNNs.
• A need exists for methods that specifically assess task-relevant decision strategies, not just general representational alignment.

Purpose of the Study:

• To introduce and validate a novel metric, decision variable correlation (DVC), for comparing decision strategies between observers (brains or models).
• To investigate the task-relevant representational similarity between monkey visual cortex (V4/IT) and DNNs trained for image classification.
• To assess how factors like network performance, adversarial training, and dataset size affect this similarity.

Main Methods:

• Developed decision variable correlation (DVC) to quantify image-by-image correlation of decoded decisions from internal representations.
• Collected neural recordings from monkey V4/IT during a classification task.
• Utilized various deep neural network models trained on image classification tasks, including those with adversarial training and large-scale pre-training.

Main Results:

• Model-model and monkey-monkey decision strategy similarities were comparable.
• Model-monkey decision strategy similarity was consistently lower than both model-model and monkey-monkey similarities.
• Decision variable correlation (DVC) decreased as network performance on ImageNet-1k increased.
• Adversarial training and large-scale pre-training did not enhance model-monkey similarity in task-relevant dimensions.

Conclusions:

• Decision variable correlation (DVC) effectively captures task-relevant information, offering a new perspective on comparing decision strategies.
• Task-relevant representations in monkey V4/IT diverge from those learned by standard image classification DNNs.
• Current training paradigms for DNNs do not fully bridge the gap in decision-making strategies compared to biological visual systems.