Quantifying task-relevant representational similarity using decision variable correlation

Area of Science:

• Neuroscience
• Artificial Intelligence
• Computer Vision

Background:

• Comparing neural representations in the visual cortex to deep neural networks (DNNs) is crucial for understanding both biological and artificial vision.
• Previous studies show mixed results regarding the similarity between neural activities and DNN representations.
• A new method is needed to specifically assess task-relevant decision strategies, not just general representational alignment.

Purpose of the Study:

• To propose and evaluate decision variable correlation (DVC) as a novel approach to quantify the similarity of decision strategies between observers (brains or models).
• To compare the task-relevant representations of monkey visual cortex (V4/IT) with those of DNNs trained on image classification.

Main Methods:

• Developed Decision Variable Correlation (DVC) to measure image-by-image correlation of decoded decisions from internal representations.
• Collected neural recordings from monkey V4/IT during a classification task.
• Utilized various DNNs trained on image classification tasks, including those with adversarial training and large-dataset pre-training.

Main Results:

• Model-model and monkey-monkey similarity were comparable, but model-monkey similarity was consistently lower.
• Decision variable correlation (DVC) decreased as network performance on ImageNet-1k increased.
• Adversarial training and large-dataset pre-training did not improve model-monkey similarity in task-relevant dimensions.

Conclusions:

• Decision variable correlation (DVC) effectively captures task-relevant information, revealing differences in decision strategies.
• Task-relevant representations in monkey V4/IT diverge from those learned by standard image classification DNNs.
• Current DNN training methods do not fully bridge the gap in decision-making strategies compared to biological vision.