High-Level Visual Encoding Model Framework with Hierarchical Ventral Stream-Optimized Neural Networks

Area of Science:

• Neuroscience
• Computational Neuroscience
• Computer Vision

Background:

• Deep neural network (DNN) based visual encoding models excel in low-level visual areas but struggle with high-level areas due to limited neural data.
• Current models do not fully capture the ventral stream's hierarchical information flow from lower to higher visual areas.

Purpose of the Study:

• To propose a novel visual encoding model framework utilizing the ventral stream's representational hierarchy.
• To improve encoding model performance in high-level visual areas like V4 and LO.

Main Methods:

• Developed two categories of hierarchical encoding models: voxel-to-voxel and feature-to-voxel.
• Voxel perspective: Modeled low-level visual areas (V1/V2) and used their predicted voxel space to predict high-level areas (V4/LO).
• Feature perspective: Extracted feature space from an initial model to predict high-level visual area voxel space.

Main Results:

• Both hierarchical encoding models significantly improved encoding performance in V4 and LO.
• Proposed models demonstrated superior prediction accuracy compared to existing models.
• The hierarchy of representations in the ventral stream positively impacts performance in high-level visual areas.

Conclusions:

• Hierarchical encoding models effectively enhance prediction accuracy in high-level visual areas.
• Leveraging the ventral stream's representational hierarchy is a promising strategy for improving visual encoding models.
• This framework addresses limitations of current DNN-based models in complex visual processing.