Characterizing Universal Object Representations Across Vision Models

Area of Science:

• Computer Vision
• Computational Neuroscience
• Artificial Intelligence

Background:

• Deep neural networks (DNNs) trained on diverse datasets often develop similar visual representations.
• However, the specific visual properties and underlying factors driving this convergence remain largely unknown.

Purpose of the Study:

• To investigate which visual properties DNNs converge on.
• To identify universal versus model-specific dimensions in DNN representations.
• To explore factors influencing the emergence of universal dimensions and their relation to biological vision.

Main Methods:

• Decomposed the object similarity structure of 162 diverse vision models.
• Analyzed the reappearance frequency of dimensions across models to identify universal dimensions.
• Correlated universal dimensions with conceptual image properties, macaque IT activity, and human similarity judgments.

Main Results:

• Identified a small set of universal, interpretable dimensions in DNNs, driven by conceptual image properties.
• Found that model architecture, objective, data, size, or performance did not explain universal dimensions.
• Models with more universal dimensions better predicted biological vision data (macaque IT activity and human judgments).

Conclusions:

• Universality in DNN representations is driven by semantic content, not just training specifics.
• Universal dimensions reflect representations aligned with biological vision.
• Findings advance understanding of emergent representations in DNNs and their biological relevance.