Representing Multiple Visual Objects in the Human Brain and Convolutional Neural Networks.

Area of Science:

• Neuroscience
• Computer Vision
• Cognitive Science

Background:

• Real-world object recognition involves processing objects within complex scenes.
• Primate studies show neural responses to object pairs approximate the average of individual object responses, suggesting 'the whole is equal to the average of its parts'.

Approach:

• Investigated averaging in human lateral occipital (LO) complex using functional magnetic resonance imaging (fMRI) at both single voxel and population levels.
• Examined five convolutional neural networks (CNNs) with varied architectures, depths, and recurrent processing to assess averaging mechanisms.

Key Points:

• Averaging was confirmed in single fMRI voxels and population responses within the human LO complex, with better single-voxel averaging correlating with population-level averaging.
• CNNs exhibited averaging at the unit level, but this rarely translated to the population level.
• CNN unit response distributions generally did not mirror those observed in human LO or macaque inferotemporal (IT) cortex.

Conclusions:

• The human brain employs an averaging mechanism for object processing, enabling robust object identification even in cluttered scenes.
• CNNs, while capable of unit-level averaging, do not fully replicate the population-level averaging observed in biological vision systems.
• Differences in averaging mechanisms may limit the accessibility of individual objects within CNNs compared to the human brain.