Representational constraints underlying similarity between task-optimized neural systems

Neural systems, artificial and biological, show similar representations of inputs when optimized to perform similar tasks. In visual systems optimized for tasks similar to object recognition, we propose that representation similarities arise from the constraints imposed by the development of abstractions in the representation across the processing stages. To study the effect of abstraction hierarchy of representations across different visual systems, we constructed a two-dimensional space in which each neural representation is positioned based on its distance from the pixel space and the class space. Trajectories of representations in all the task-optimized visual neural networks start close to the pixel space and gradually move towards higher abstract representations, such as object categories. We also observe that proximity in this abstraction space predicts the similarity of neural representations between visual systems. The gradual similar change of the representations suggests that the similarity across different task-optimized systems could arise from constraints on representational trajectories.