Multimodal neural networks better explain multivoxel patterns in the hippocampus

Area of Science:

• Neuroscience
• Artificial Intelligence
• Cognitive Science

Background:

• The human hippocampus contains "concept cells" that respond to specific concepts across different sensory inputs.
• Recent research identified similar concept cells in the multimodal network CLIP (Contrastive Language-Image Pre-training).

Purpose of the Study:

• To investigate whether CLIP can explain human hippocampus fMRI activity more effectively than unimodal models.
• To evaluate a range of uni- and multimodal models for their explanatory power regarding hippocampal function.

Main Methods:

• Comparative analysis of fMRI data from the human hippocampus.
• Assessment of various artificial intelligence models, including unimodal (visual, linguistic) and multimodal networks.
• Evaluation of model performance in predicting multivoxel activity patterns in the hippocampus.

Main Results:

• Multimodal models, particularly CLIP, demonstrated a superior ability to explain hippocampal fMRI activity compared to unimodal models.
• The study confirmed that "multimodality" is a critical factor in explaining neural representations of concepts.
• CLIP's architecture showed significant promise in modeling brain activity related to conceptual understanding.

Conclusions:

• Multimodal artificial intelligence networks offer a powerful framework for understanding the neural basis of concept representation in the human hippocampus.
• The findings suggest that integrating information across different modalities is fundamental to how the brain processes and represents concepts.
• Future research can leverage multimodal AI to further explore cognitive functions associated with the hippocampus.