Brain-guided convolutional neural networks reveal task-specific representations in scene processing.

Area of Science:

• Neuroscience
• Computer Vision
• Cognitive Science

Background:

• Human visual system processes complex scenes for diverse tasks beyond simple categorization.
• Understanding how task demands alter neural representations of visual information is crucial.
• Current models lack insight into the dynamic, task-specific feature utilization in the brain.

Purpose of the Study:

• To develop a novel brain-guided convolutional neural network (CNN) that integrates neural data to understand task-specific visual processing.
• To investigate how different visual tasks (object detection vs. scene affordance) modulate the spatiotemporal use of image features.
• To spatially assess feature utilization across CNN layers guided by human neural responses.

Main Methods:

• A novel brain-guided CNN was developed, with each layer informed by neural responses from human observers performing object detection or scene affordance tasks.
• Deconvolution techniques were used to reconstruct and analyze activation maps from each CNN layer.
• Neural data was collected while participants viewed the same set of images for different cued tasks.

Main Results:

• The brain-guided CNN successfully utilized image features critical for task completion, aligning with human observer data between 244 ms and 402 ms.
• Analysis of activation maps demonstrated that the CNN, guided by neural data, learned task-relevant differences in feature representation.
• Systematic evolution of spatiotemporal representations of local image features across CNN layers was observed, reflecting task-specific processing.

Conclusions:

• The brain's representation of visual information is dynamic and systematically evolves over time, influenced by task goals.
• Distinct image features are processed and utilized at different stages of neural processing, shaped by behavioral context.
• Brain-guided CNNs offer a powerful tool for dissecting the neural mechanisms underlying task-specific visual perception.