A Comparative Analysis of Visual Encoding Models Based on Classification and Segmentation Task-Driven CNNs

Area of Science:

• Neuroscience
• Computer Vision
• Machine Learning

Background:

• Convolutional neural networks (CNNs) are widely used in computer vision to model human information processing.
• Different network tasks can lead to variations in the predictive performance of visual encoding models.
• Understanding how network tasks influence these models is crucial for advancing artificial intelligence and neuroscience.

Purpose of the Study:

• To investigate the impact of different network tasks on the performance of visual encoding models.
• To compare the effectiveness of segmentation and classification networks in predicting brain responses.
• To determine which type of task-based network more closely mimics human visual processing.

Main Methods:

• Utilized functional magnetic resonance imaging (fMRI) data from natural visual stimulation.
• Extracted features using a segmentation network (FCN32s) and a classification network (VGG16).
• Employed regularized orthogonal matching pursuit (ROMP) to map extracted features to voxel responses, using segmentation, classification, and fused features.

Main Results:

• Encoding models based on different network tasks effectively predicted stimulus-induced fMRI responses, but with varying accuracy.
• The VGG16 (classification) model showed significantly higher prediction accuracy than the FCN32s (segmentation) model across most voxels.
• Prediction accuracy for the VGG16 model was comparable to that of fused features.

Conclusions:

• CNNs performing classification tasks demonstrate a closer resemblance to human visual processing compared to those performing segmentation tasks.
• The choice of task for a CNN significantly influences its ability to serve as a model for human visual encoding.
• These findings have implications for developing more accurate brain-inspired AI systems and understanding neural representations.