Using deep neural networks to disentangle visual and semantic information in human perception and memory.

Area of Science:

• Cognitive Neuroscience
• Computational Neuroscience
• Psychology

Background:

• Mental representations integrate visual and semantic information.
• Disentangling these components in human cognition is challenging due to their intermixing.
• Deep neural networks offer novel methods to analyze these components.

Purpose of the Study:

• To disentangle visual, semantic, and integrated visual-semantic components of human mental representations.
• To investigate the content of mental representations for familiar faces and objects during perception and memory recall.
• To propose a new framework for understanding the interplay of visual and semantic information in cognition.

Main Methods:

• Utilized deep neural networks trained on images, text, or paired image-text data.
• Analyzed mental representations of familiar faces and objects.
• Compared representations during visual perception versus memory recall.

Main Results:

• Perception of images showed a greater visual contribution than semantic.
• Memory recall of images revealed a reversed pattern, with greater semantic contribution.
• Identified a unique, integrated visual-semantic representation component in both perception and memory.

Conclusions:

• Visual and semantic information contribute distinctly to mental representations.
• A novel integrated visual-semantic component plays a crucial role in both perception and memory.
• Proposed a framework where visual and semantic information interact dynamically in mental representations.