Investigating the temporal dynamics and modeling of mid-level feature representations in humans

Area of Science:

• Neuroscience
• Computer Vision
• Cognitive Science

Background:

• Visual perception involves a hierarchy from low-level (edges) to high-level (object categories) features.
• The processing of intermediate, or mid-level, visual features remains poorly understood.
• Understanding mid-level vision is key to bridging sensory input and semantic understanding.

Purpose of the Study:

• To investigate the timing and nature of mid-level visual feature processing in the human brain.
• To introduce and validate a novel stimulus set of 3D-rendered images and videos for studying mid-level vision.
• To compare human mid-level visual processing with convolutional neural network (CNN) models.

Main Methods:

• Developed a stimulus set of 3D-rendered naturalistic images and videos with annotations for low-, mid-, and high-level features.
• Recorded electroencephalography (EEG) responses during stimulus presentation.
• Trained linearized encoding models to predict EEG responses from stimulus annotations.

Main Results:

• Mid-level features (reflectance, depth, normals, lighting, skeleton) were processed between approximately 100 and 250 ms post-stimulus.
• This timing places mid-level feature processing between low- and high-level features, suggesting a bridging role.
• CNNs showed similar processing order for mid-level features in videos, but not for low- or high-level features, compared to humans.

Conclusions:

• Mid-level visual features are critical for surface and shape processing, linking sensory and semantic stages.
• The 3D-rendered stimulus set is a valuable tool for future research on mid-level vision in both biological and artificial systems.
• The findings provide insights into the hierarchical organization of the human visual system and the capabilities of CNNs.